Planet Maemo: category "feed:01c9bb92c8cdc7e3a5fc627f123acc22" Blog entries from Maemo community http://maemo.org/news/planet-maemo/ Sun, 24 May 2026 11:08:25 +0000 FeedCreator 1.7.6(BH) en planet@maemo.org Eina_Log customizations http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/kt23Orc93WI/ Introduction to Eina_Log:

EFL provides a powerful logging system called “eina_log”, it can help developers and QA engineers a lot by providing key features:

  • Levels/Priorities with option to disable at compile time: similar to syslog, one can specify levels such as critical, error, warning, information or debug. An added bonus is that you can compile-out messages above certain levels, removing the function call overhead and the strings from the binary.
  • Logging domains: one can create multiple domains aside from the global/default one, this allows for easy filtering when debugging or looking for errors. Let’s say you want debug Edje, just export EINA_LOG_LEVELS="edje:4"
  • Comprehensive, Colorful default formatting: it will automatically turn on colors if your $TERM is supported (also works on Windows). Output includes PID, log domain, file name, line number and even thread ID if that’s different from the main thread. Some of the components can be toggled with environment variables.
The benefits of writing with eina_log instead of adding printf() is that you can more easily filter the output and you can keep the debug information there, without extra impacts. Often people that add printf() will forget them and pollute everybody’s else output, eventually clashing with user-output (if printf()/stdout and not fprintf()/stderr). Or they remove and when someone else run into similar bugs and needs to walk the code and think all over again on where to add logs to figure out the problem. Then it’s best that the developer who writes the code leaves proper debug messages with key variables/values, for future debug help.

Although widely documented and with examples at http://docs.enlightenment.org/auto/eina/group__Eina__Log__Group.html some still don’t use it due lack of knowledge, particularly on how to customize the output. They end writing new systems, such as Tizen did with “dlog“.

Customizing Eina_Log output:

The application can specify its own logging function with eina eina_log_print_cb_set(function, function_data). By default it will use eina_log_print_cb_stderr(), but eina_log_print_cb_stdout() and eina_log_print_cb_file() are available. We’ll explore the options by writing our own functions (all source hosted at http://barbieri-playground.googlecode.com/svn/efl-tests/eina-log/)

Simple Log to Stderr:

Let’s start with a simple test that just prints to stderr without fancy formatting. Each line is the log level followed by the actual message:

#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    fprintf(stderr, "%d: ", level);
    vfprintf(stderr, fmt, args);
    putc('\n', stderr);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output:

$ gcc -o eina-log-simple eina-log-simple.c `pkg-config --cflags --libs eina`
$ ./eina-log-simple
1: Hi there: 1234

It starts as usual initializing Eina, then sets our function to be used as print callback, then uses the standard macro that logs to the global domain. Our function is also pretty simple and easy to understand.

Simple Log just our domain to Stderr:

How about if we just want the simple output for our application domain, leaving all others with Eina’s standard eina_log_print_cb_stderr()? This was required by edje_cc tool. It’s just about using the const Eina_Log_Domain *d parameter!

#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
   if ((!d->name) || (strcmp(d->name, "mydomain") != 0))
     eina_log_print_cb_stderr(d, level, file, fnc, line, fmt, NULL, args);
   else
     {
        fprintf(stderr, "%d: ", level);
        vfprintf(stderr, fmt, args);
        putc('\n', stderr);
     }
}

int main(int argc, char *argv[])
{
   int log_domain;

   eina_init();
   eina_log_print_cb_set(_my_log_cb, NULL);

   log_domain = eina_log_domain_register("mydomain", NULL);

   EINA_LOG_ERR("Hi there: %d", 1234);
   EINA_LOG_DOM_ERR(log_domain, "Just for domain: %x", 0xff00);

   eina_shutdown();
   return 0;
}

Output:

$ ./eina-log-simple-domain
ERR: eina-log-simple-domain.c:34 main() Hi there: 1234
1: Just for domain: ff00

As one can see, just “mydomain” is printed in a simpler way, other domains are still using Eina’s standard output.

Log to Syslog:

If you’re writing a key component of some platform, such as the phone dialer, you may want to log it to syslog. It may also be useful in some embedded systems. That’s as simple:

#include <Eina.h>
#include <syslog.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }
    vsyslog(priority, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output:

Oct 27 16:20:18 solid eina-log-syslog[25582]: Hi there: 1234

As simple as you wondered, just noticed that we are converting Eina’s level to syslog priorities.

Log to Systemd’s Journal

Systemd will be used in Tizen, then it would be nice to have Eina to log to its powerful Journal system. Lennart wrote a nice article systemd for Developers III where he explains how to do it and the benefits. Looking at sd-journal.h we can see how to do it:

#include <Eina.h>
#include <systemd/sd-journal.h>

/* http://0pointer.de/blog/projects/journal-submit */
/* gcc -o test test.c `pkg-config --libs --cflags eina libsystemd-journal` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    char filestr[PATH_MAX + sizeof("CODE_FILE=")];
    char linestr[128];
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }

    snprintf(filestr, sizeof(filestr), "CODE_FILE=%s", file);
    snprintf(linestr, sizeof(linestr), "CODE_LINE=%d", line);
    sd_journal_printv_with_location(priority, filestr, linestr, fnc, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}

Output (json-pretty)

$ ./eina-log-journal
$ journalctl -o json-pretty
{
        "__CURSOR" : "s=3c7eaf66e4734bb286e54efd20bb9f8d;i=645;b=c6681f0bd53d41d4ba682e12021095e7;m=23487887b;t=4cd0e8d5c0bf3;x=ab7f934fdb40485f;p=system.journal",
        "__REALTIME_TIMESTAMP" : "1351362291698675",
        "__MONOTONIC_TIMESTAMP" : "9471232123",
        "_BOOT_ID" : "c6681f0bd53d41d4ba682e12021095e7",
        "MESSAGE" : "Hi there: 1234",
        "PRIORITY" : "3",
        "CODE_FILE" : "eina-log-journal.c",
        "CODE_LINE" : "50",
        "CODE_FUNC" : "main",
        "_TRANSPORT" : "journal",
        "_PID" : "25655",
        "_UID" : "500",
        "_GID" : "500",
        "_SOURCE_REALTIME_TIMESTAMP" : "1351362291698325",
        "_MACHINE_ID" : "b84a5d286a7bfe411836d69100004cf6",
        "_HOSTNAME" : "solid"
}

See that we can get our file, line and function to be saved by the journal. Just awesome! :-)

4 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Sat, 27 Oct 2012 18:33:27 +0000 http://maemo.org/midcom-permalink-f99e9b54206411e29d8c892346603ccf3ccf Eina_Log customizations http://blog.gustavobarbieri.com.br/2012/10/27/eina_log-customizations.html Introduction to Eina_Log: EFL provides a powerful logging system called "eina_log", it can help developers and QA engineers a lot by providing key features:
  • Levels/Priorities with option to disable at compile time: similar to syslog, one can specify levels such as critical, error, warning, information or debug. An added bonus is that you can compile-out messages above certain levels, removing the function call overhead and the strings from the binary.
  • Logging domains: one can create multiple domains aside from the global/default one, this allows for easy filtering when debugging or looking for errors. Let's say you want debug Edje, just export EINA_LOG_LEVELS="edje:4"
  • Comprehensive, Colorful default formatting: it will automatically turn on colors if your $TERM is supported (also works on Windows). Output includes PID, log domain, file name, line number and even thread ID if that's different from the main thread. Some of the components can be toggled with environment variables.
The benefits of writing with eina_log instead of adding printf() is that you can more easily filter the output and you can keep the debug information there, without extra impacts. Often people that add printf() will forget them and pollute everybody's else output, eventually clashing with user-output (if printf()/stdout and not fprintf()/stderr). Or they remove and when someone else run into similar bugs and needs to walk the code and think all over again on where to add logs to figure out the problem. Then it's best that the developer who writes the code leaves proper debug messages with key variables/values, for future debug help.
Although widely documented and with examples at http://docs.enlightenment.org/auto/eina/group__Eina__Log__Group.html some still don't use it due lack of knowledge, particularly on how to customize the output. They end writing new systems, such as Tizen did with "dlog". Customizing Eina_Log output: The application can specify its own logging function with eina eina_log_print_cb_set(function, function_data). By default it will use eina_log_print_cb_stderr(), but eina_log_print_cb_stdout() and eina_log_print_cb_file() are available. We'll explore the options by writing our own functions (all source hosted at http://barbieri-playground.googlecode.com/svn/efl-tests/eina-log/) Simple Log to Stderr: Let's start with a simple test that just prints to stderr without fancy formatting. Each line is the log level followed by the actual message: 
#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    fprintf(stderr, "%d: ", level);
    vfprintf(stderr, fmt, args);
    putc('\n', stderr);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}
Output: 
$ gcc -o eina-log-simple eina-log-simple.c `pkg-config --cflags --libs eina`
$ ./eina-log-simple
1: Hi there: 1234
It starts as usual initializing Eina, then sets our function to be used as print callback, then uses the standard macro that logs to the global domain. Our function is also pretty simple and easy to understand. Simple Log just our domain to Stderr: How about if we just want the simple output for our application domain, leaving all others with Eina's standard eina_log_print_cb_stderr()? This was required by edje_cc tool. It's just about using the const Eina_Log_Domain *d parameter! 
#include <Eina.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
   if ((!d->name) || (strcmp(d->name, "mydomain") != 0))
     eina_log_print_cb_stderr(d, level, file, fnc, line, fmt, NULL, args);
   else
     {
        fprintf(stderr, "%d: ", level);
        vfprintf(stderr, fmt, args);
        putc('\n', stderr);
     }
}

int main(int argc, char *argv[])
{
   int log_domain;

   eina_init();
   eina_log_print_cb_set(_my_log_cb, NULL);

   log_domain = eina_log_domain_register("mydomain", NULL);

   EINA_LOG_ERR("Hi there: %d", 1234);
   EINA_LOG_DOM_ERR(log_domain, "Just for domain: %x", 0xff00);

   eina_shutdown();
   return 0;
}
Output: 
$ ./eina-log-simple-domain
ERR: eina-log-simple-domain.c:34 main() Hi there: 1234
1: Just for domain: ff00
As one can see, just "mydomain" is printed in a simpler way, other domains are still using Eina's standard output. Log to Syslog: If you're writing a key component of some platform, such as the phone dialer, you may want to log it to syslog. It may also be useful in some embedded systems. That's as simple: 
#include <Eina.h>
#include <syslog.h>

/* gcc -o test test.c `pkg-config --libs --cflags eina` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }
    vsyslog(priority, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}
Output: 
Oct 27 16:20:18 solid eina-log-syslog[25582]: Hi there: 1234
As simple as you wondered, just noticed that we are converting Eina's level to syslog priorities. Log to Systemd's Journal Systemd will be used in Tizen, then it would be nice to have Eina to log to its powerful Journal system. Lennart wrote a nice article systemd for Developers III where he explains how to do it and the benefits. Looking at sd-journal.h we can see how to do it: 
#include <Eina.h>
#include <systemd/sd-journal.h>

/* http://0pointer.de/blog/projects/journal-submit */
/* gcc -o test test.c `pkg-config --libs --cflags eina libsystemd-journal` */

static void
_my_log_cb(const Eina_Log_Domain *d,
           Eina_Log_Level level,
           const char *file,
           const char *fnc,
           int line,
           const char *fmt,
           void *data,
           va_list args)
{
    char filestr[PATH_MAX + sizeof("CODE_FILE=")];
    char linestr[128];
    int priority;
    switch (level) {
     case EINA_LOG_LEVEL_CRITICAL:
        priority = LOG_CRIT;
        break;
     case EINA_LOG_LEVEL_ERR:
        priority = LOG_ERR;
        break;
     case EINA_LOG_LEVEL_WARN:
        priority = LOG_WARNING;
        break;
     case EINA_LOG_LEVEL_INFO:
        priority = LOG_INFO;
        break;
     case EINA_LOG_LEVEL_DBG:
        priority = LOG_DEBUG;
        break;
     default:
        priority = level + LOG_CRIT;
    }

    snprintf(filestr, sizeof(filestr), "CODE_FILE=%s", file);
    snprintf(linestr, sizeof(linestr), "CODE_LINE=%d", line);
    sd_journal_printv_with_location(priority, filestr, linestr, fnc, fmt, args);
}

int main(int argc, char *argv[])
{
    eina_init();
    eina_log_print_cb_set(_my_log_cb, NULL);

    EINA_LOG_ERR("Hi there: %d", 1234);

    eina_shutdown();
    return 0;
}
Output (json-pretty) 
$ ./eina-log-journal
$ journalctl -o json-pretty
{
        "__CURSOR" : "s=3c7eaf66e4734bb286e54efd20bb9f8d;i=645;b=c6681f0bd53d41d4ba682e12021095e7;m=23487887b;t=4cd0e8d5c0bf3;x=ab7f934fdb40485f;p=system.journal",
        "__REALTIME_TIMESTAMP" : "1351362291698675",
        "__MONOTONIC_TIMESTAMP" : "9471232123",
        "_BOOT_ID" : "c6681f0bd53d41d4ba682e12021095e7",
        "MESSAGE" : "Hi there: 1234",
        "PRIORITY" : "3",
        "CODE_FILE" : "eina-log-journal.c",
        "CODE_LINE" : "50",
        "CODE_FUNC" : "main",
        "_TRANSPORT" : "journal",
        "_PID" : "25655",
        "_UID" : "500",
        "_GID" : "500",
        "_SOURCE_REALTIME_TIMESTAMP" : "1351362291698325",
        "_MACHINE_ID" : "b84a5d286a7bfe411836d69100004cf6",
        "_HOSTNAME" : "solid"
}
See that we can get our file, line and function to be saved by the journal. Just awesome! :-)0 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Sat, 27 Oct 2012 18:33:00 +0000 http://maemo.org/midcom-permalink-1e71f95b9c6d5ac1f9511e799c74d55c3ebdfbddfbd econnman-1 released http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/QiQiwYjQtkY/ Today I’m releasing the first release of “econnman”, the EFL user interface for ConnMan.

econnman-services econnman-service-detail econnman-tech-list econnman-tech-detail

Some time ago raster added a ConnMan module to e17, but then the API broken during 0.5 transition and I rewrote that module. With ConnMan-1.0 the API was changed yet again and our module stopped working.

Lucas De Marchi took the task to revive the module and he is still finishing this task… let’s say Enlightenment’s internal widget set (e_widget) is something that can get everyone bored to death! Also the new ConnMan API requires an Agent interface to reply with passwords, usernames and other input that the manager may need. This code was not written before, so these new API and GUI must be written.

Until Lucas can finish his work I took a weekend to play with my favorite language: Python. I was away from Python-EFL for a while, so I was wanting to write some code using it. Python-DBus is super-simple (compare it to C!), so the EConnMan in Python-EFL was a nice thing to play.

We’ll cooperate efforts: E17 module will only provide the services list and allow to connect and disconnect. It will also provide an Agent, so we can provide passwords. Everything else that is related to configuration will be left to stand-alone “econnman” application. This standalone will do the tweaks and show details such as IP and MAC addresses, proxy, tethering and others.

In less than two days I had the base of the code I’m releasing now. It supports everything I need (IOW: it does not support VPN, Bluetooth-PAN or 3G/Cellular), but it’s easy to extend to support new features…. so patches are welcome!

My delay to have it released was that I wanted to improve theme (both default and detourious) so it look better, as you can see in the screenshots. It’s still not perfect, particularly the “Segment Control” used to show mutually exclusive options (DHCP, Static, Off) is damn ugly!

It also ships with an optional agent (–agent/-a command line option) that can be used until E17 gets a proper module with agent support.

Download:

Future:

Some people want IPv6, Proxy and VPN support. Those are nice ideas, but I personally don’t need them. So I’d like some people that use them to step up and do the code… or at least push me to do them! :-P

Some people want to rewrite it in C and while I can sponsor the idea, it will be much more work for no real gain. With a Python version working one can do the conversion easily, even the helper structures and callbacks can be known beforehand…

 

7 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Mon, 13 Aug 2012 00:11:49 +0000 http://maemo.org/midcom-permalink-bac81458e4de11e196d39b68c9a646d046d0 econnman-1 released http://blog.gustavobarbieri.com.br/2012/08/13/econnman-1-released.html EFL user interface for ConnMan.
Some time ago raster added a ConnMan module to e17, but then the API broken during 0.5 transition and I rewrote that module. With ConnMan-1.0 the API was changed yet again and our module stopped working. Lucas De Marchi took the task to revive the module and he is still finishing this task... let's say Enlightenment's internal widget set (e_widget) is something that can get everyone bored to death! Also the new ConnMan API requires an Agent interface to reply with passwords, usernames and other input that the manager may need. This code was not written before, so these new API and GUI must be written. Until Lucas can finish his work I took a weekend to play with my favorite language: Python. I was away from Python-EFL for a while, so I was wanting to write some code using it. Python-DBus is super-simple (compare it to C!), so the EConnMan in Python-EFL was a nice thing to play. We'll cooperate efforts: E17 module will only provide the services list and allow to connect and disconnect. It will also provide an Agent, so we can provide passwords. Everything else that is related to configuration will be left to stand-alone "econnman" application. This standalone will do the tweaks and show details such as IP and MAC addresses, proxy, tethering and others. In less than two days I had the base of the code I'm releasing now. It supports everything I need (IOW: it does not support VPN, Bluetooth-PAN or 3G/Cellular), but it's easy to extend to support new features.... so patches are welcome! My delay to have it released was that I wanted to improve theme (both default and detourious) so it look better, as you can see in the screenshots. It's still not perfect, particularly the "Segment Control" used to show mutually exclusive options (DHCP, Static, Off) is damn ugly! It also ships with an optional agent (--agent/-a command line option) that can be used until E17 gets a proper module with agent support. Download: Future: Some people want IPv6, Proxy and VPN support. Those are nice ideas, but I personally don't need them. So I'd like some people that use them to step up and do the code... or at least push me to do them! :-P Some people want to rewrite it in C and while I can sponsor the idea, it will be much more work for no real gain. With a Python version working one can do the conversion easily, even the helper structures and callbacks can be known beforehand...  0 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Mon, 13 Aug 2012 00:11:00 +0000 http://maemo.org/midcom-permalink-1e71f95b726f8721f9511e799c74d55c3ebdfbddfbd kmod announcement (and how to help testing it!) http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/ShX2gzyHp8s/ introduction and motivation

At ProFUSION a common topic is how to optimze the system. Not just the speed, but also what and how it is done. Not just for our embedded systems, but also our desktops and laptops.

These discussions led us to be involved in projects that consider this goal, such as Enlightenment, ConnMan, oFono and systemd. With great projects, come great people and thus enlightening discussions on how to improve things. From these discussions Lennart Poettering and Kay Sievers put together a A Plumber’s Wish List for Linux, with one of the items being of special interest for ProFUSION as it had the potential of also helping embedded systems and speeding up or boots:

* module-init-tools: provide a proper libmodprobe.so from
module-init-tools:
Early boot tools, installers, driver install disks want to
access information about available modules to optimize
bootup handling.

One of the reasons for this is that udev will search /sys/devices for all “modalias” files and call “/sbin/modprobe -bv $ALIAS“, however many of these calls evaluate to an empty list, thus an useless execution of program that will open /lib/modules/`uname -r`/modules.*, load resources, search for something that is not there and return. This could be way cheaper if done inside udev workers. [note: Lucas did measure and noticed 2.5x speedups, stay tuned for benchmarks]

kmod

Then our developer Lucas De Marchi proposed to do the libmodprobe and we funded it. After some discussions with Lennart and Kay, it was decided to rename it to kmod for shortness and good namespace. The initial goal was to achive a 1:1 replacement of module-init-tools as a library to be used by udev and systemd-modules-load and we are close to it with our second release done today! While we miss depmod tool (planned for v3), our kmod-modprobe should be fully functional and if this proves to be true, the logic will move to libkmod to be used by udev.

we need your testing!

To ensure kmod-modprobe does what it is supposed to do, we need extensive testing, there is where we need you! Try it on as many systems as possibles and let us know. To do this we recommend:

$ git clone git://git.profusion.mobi/kmod.git
$ cd kmod
$ ./autogen.sh
$ ./configure --enable-logging \
              --enable-debug --enable-tools \
              --enable-zlib # if you have module.ko.gz
$ make all
$ sudo make install
$ sudo mv /sbin/modprobe /sbin/modprobe-bkp
$ sudo ln -s /usr/bin/kmod-modprobe /sbin/modprobe
$ ls -1 /sys/module/ > ~/original-modules
$ reboot
$ ls -1 /sys/module/ > ~/kmod-modules
$ diff ~/original-modules ~/kmod-modules

current users and feedback

Being a new project in a critical area of Linux system, we expected lots of criticism and rejection by people, but interestingly enough after Lucas’ announcement and LWN article the feedback was highly positive! We even had some testers and people to help with ideas and experience.

Among the people that joined the project is the current maintainer of module-init-tools package Jon Masters, which announced that kmod replaces module-init-tools. He is helping a lot with his knowledge and cases from Red Hat Enterprise Linux, also providing modules from non-trivial platforms such as s390. Last but not least he provided insights to improve module handling on Linux, particularly replacing modules with better alternative as required in enterprise systems.

With Jon’s blessing we’ll use the same mailing list linux-modules@vger.kernel.org and our git will move to kernel.org soon.

People from ARCH and Debian were also interested and even created packages for it! These guys were extremely helpful to test cases such as gzip modules, 64bits modules with i386 user space and so on. They are: Tom Gundersen (tomegun), Dave Reisner (falconindy) and Marco d’Itri (Md).

next steps

Based on our TODO, we have the following ideas for next steps:

  • Finish libkmod-elf.c to provide information required by depmod. I’m working on this in my depmod branch;
  • tools/kmod-depmod.c: create a 1:1 compatible tool to generate /lib/modules/`uname -r`/modules.*;
  • libkmod should export parsed configuration to be used by kmod-modprobe.c –dumpconfig;
  • create kmodmock library to be LD_PRELOAD’ed to redirect some syscalls such as init_module(), delete_module() and open() of /sys and /proc. I’ve started a branch for this some time ago as “unittest” branch but stopped due lack of ELF support at the time;
  • create unittests and measure coverage. Given kernel modules are user-input they may be broken files and libkmod-elf.c needs to be extensively tested to avoid segmentation fault due out-of-boundaries access. This mean not trusting null terminated string in .strtab section and so on.
  • create optimized modules.archive that would contain optimized search index and all modules compressed independently, but in the same file. This would save file access and could provide information we don’t have today, like the uncompressed size of gzipped modules.
7 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Wed, 21 Dec 2011 18:23:19 +0000 http://maemo.org/midcom-permalink-7f2db5aa2c0211e1be3477741e8fcce1cce1 kmod announcement (and how to help testing it!) http://blog.gustavobarbieri.com.br/2011/12/21/kmod-announcement-and-how-to-help-testing-it.html introduction and motivation At ProFUSION a common topic is how to optimze the system. Not just the speed, but also what and how it is done. Not just for our embedded systems, but also our desktops and laptops. These discussions led us to be involved in projects that consider this goal, such as Enlightenment, ConnMan, oFono and systemd. With great projects, come great people and thus enlightening discussions on how to improve things. From these discussions Lennart Poettering and Kay Sievers put together a A Plumber's Wish List for Linux, with one of the items being of special interest for ProFUSION as it had the potential of also helping embedded systems and speeding up or boots: 
* module-init-tools: provide a proper libmodprobe.so from
module-init-tools:
Early boot tools, installers, driver install disks want to
access information about available modules to optimize
bootup handling.
One of the reasons for this is that udev will search /sys/devices for all "modalias" files and call "/sbin/modprobe -bv $ALIAS", however many of these calls evaluate to an empty list, thus an useless execution of program that will open /lib/modules/`uname -r`/modules.*, load resources, search for something that is not there and return. This could be way cheaper if done inside udev workers. [note: Lucas did measure and noticed 2.5x speedups, stay tuned for benchmarks]

kmod

Then our developer Lucas De Marchi proposed to do the libmodprobe and we funded it. After some discussions with Lennart and Kay, it was decided to rename it to kmod for shortness and good namespace. The initial goal was to achive a 1:1 replacement of module-init-tools as a library to be used by udev and systemd-modules-load and we are close to it with our second release done today! While we miss depmod tool (planned for v3), our kmod-modprobe should be fully functional and if this proves to be true, the logic will move to libkmod to be used by udev.

we need your testing!

To ensure kmod-modprobe does what it is supposed to do, we need extensive testing, there is where we need you! Try it on as many systems as possibles and let us know. To do this we recommend: 
$ git clone git://git.profusion.mobi/kmod.git
$ cd kmod
$ ./autogen.sh
$ ./configure --enable-logging \
              --enable-debug --enable-tools \
              --enable-zlib # if you have module.ko.gz
$ make all
$ sudo make install
$ sudo mv /sbin/modprobe /sbin/modprobe-bkp
$ sudo ln -s /usr/bin/kmod-modprobe /sbin/modprobe
$ ls -1 /sys/module/ > ~/original-modules
$ reboot
$ ls -1 /sys/module/ > ~/kmod-modules
$ diff ~/original-modules ~/kmod-modules

current users and feedback

Being a new project in a critical area of Linux system, we expected lots of criticism and rejection by people, but interestingly enough after Lucas' announcement and LWN article the feedback was highly positive! We even had some testers and people to help with ideas and experience. Among the people that joined the project is the current maintainer of module-init-tools package Jon Masters, which announced that kmod replaces module-init-tools. He is helping a lot with his knowledge and cases from Red Hat Enterprise Linux, also providing modules from non-trivial platforms such as s390. Last but not least he provided insights to improve module handling on Linux, particularly replacing modules with better alternative as required in enterprise systems. With Jon's blessing we'll use the same mailing list linux-modules@vger.kernel.org and our git will move to kernel.org soon. People from ARCH and Debian were also interested and even created packages for it! These guys were extremely helpful to test cases such as gzip modules, 64bits modules with i386 user space and so on. They are: Tom Gundersen (tomegun), Dave Reisner (falconindy) and Marco d’Itri (Md).

next steps

Based on our TODO, we have the following ideas for next steps:
  • Finish libkmod-elf.c to provide information required by depmod. I'm working on this in my depmod branch;
  • tools/kmod-depmod.c: create a 1:1 compatible tool to generate /lib/modules/`uname -r`/modules.*;
  • libkmod should export parsed configuration to be used by kmod-modprobe.c --dumpconfig;
  • create kmodmock library to be LD_PRELOAD'ed to redirect some syscalls such as init_module(), delete_module() and open() of /sys and /proc. I've started a branch for this some time ago as "unittest" branch but stopped due lack of ELF support at the time;
  • create unittests and measure coverage. Given kernel modules are user-input they may be broken files and libkmod-elf.c needs to be extensively tested to avoid segmentation fault due out-of-boundaries access. This mean not trusting null terminated string in .strtab section and so on.
  • create optimized modules.archive that would contain optimized search index and all modules compressed independently, but in the same file. This would save file access and could provide information we don't have today, like the uncompressed size of gzipped modules.
0 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Wed, 21 Dec 2011 18:23:00 +0000 http://maemo.org/midcom-permalink-1e71f95b4b88b0a1f9511e799c74d55c3ebdfbddfbd PythonBrasil[7] = Excellent! http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/wNRVEWB4VXg/ Last week from 29-Sep to 01-Oct we had the amazing 7th PythonBrasil conference, for the first time in São Paulo.

Since I’ve start to use Python in 2002 I loved the language, but after getting introduced to the PythonBrasil community in 2004 I’ve boosted my development skills, got some friends and even my first job (INdT-Recife) was a kind recommendation from Osvaldo Santana in 2005.

By 2005 we had the 1st PythonBrasil Conference, then called PyConBrasil, here in Campinas with the help of UNICAMP and our amazing non-stop contributor Rodrigo Senra. It was very cool, I even presented a talk there… and it motivated me to go to following conferences in 2006 and 2007 as well.

However if starting to work at INdT reduced my spare time since late 2006, after ProFUSION was born in 2008 I had no time to participate in the lists or even go to conferences. What a shame!

I couldn’t see how shameful it was until I did this PythonBrasil in 2011. I’ m yet to see a conference with so kind people. People still remembered me and I was ashamed when I couldn’t remind their names… although they did remember mine (NOTE TO CONFERENCE: bigger names next year!) Some would even let me know they still use Eagle-Py, something that I already forgot about. And people I had closer contact before were willing to talk as if we had met last week. Amazing.

During these talks I’ve catch up with Rodrigo Senra, Luciano, Osvaldo, Erico, Marco André, Sidnei, Fernando and many more I couldn’t remember. However one of the talks was very special: talked to Gustavo Niemeyer about Go programming language. That’s right, people were so kind and open we had a keynote about Go, and we talked a lot afterwards without problems! :-P  Gustavo showed me some nice details about the language and my mind is now burning! I must do Enlightenment Foundation Libraries (EFL) + Go = EGO, a perfect GUI tool.

But I was of use… not just a leecher! I went there to present 3 talks (slides in Portuguese!):

Last but not least, I’d like to thank everyone that did this amazing conference possible!

5 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 06 Oct 2011 21:00:41 +0000 http://maemo.org/midcom-permalink-911fee84f0ae11e0995389975b483e4d3e4d PythonBrasil[7] = Excellent! http://blog.gustavobarbieri.com.br/2011/10/06/pythonbrasil7-excellent.html NOTE TO CONFERENCE: bigger names next year!) Some would even let me know they still use Eagle-Py, something that I already forgot about. And people I had closer contact before were willing to talk as if we had met last week. Amazing. During these talks I've catch up with Rodrigo Senra, Luciano, Osvaldo, Erico, Marco André, Sidnei, Fernando and many more I couldn't remember. However one of the talks was very special: talked to Gustavo Niemeyer about Go programming language. That's right, people were so kind and open we had a keynote about Go, and we talked a lot afterwards without problems! :-P Gustavo showed me some nice details about the language and my mind is now burning! I must do Enlightenment Foundation Libraries (EFL) + Go = EGO, a perfect GUI tool. But I was of use... not just a leecher! I went there to present 3 talks (slides in Portuguese!): Last but not least, I'd like to thank everyone that did this amazing conference possible!0 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 06 Oct 2011 21:00:00 +0000 http://maemo.org/midcom-permalink-1e71f95b25041a01f9511e799c74d55c3ebdfbddfbd Introducing EWS to help EFL on FB and PS3 http://blog.gustavobarbieri.com.br/2011/10/06/introducing-ews-to-help-efl-on-fb-and-ps3.html QWS, a great help for developers. Considering QWS I decided to help our PS3 friends with something similar which I shamelessly called EWS (Ecore + Evas Single Process Windowing System). Ecore and Evas already made 99% of this work: they abstract rendering, abstract windowing and so on. There is even support for "inlined windows" in Elementary, these are rendered to buffers that are visible inside other windows. Actually the first version of EWS was a 30min hack in Elementary called "eland" (making fun of wayland). But as not everyone would use Elementary it was better to move it lower in the stack: Ecore_Evas. Ecore_Evas is a glue layer between Ecore and Evas that takes care to configure actual windows and setup Evas on them, to capture events from various sources and feed them to canvas, etc. If you cope with its API you'll work with most EFL libraries and applications out-of-the-box. Ecore_Evas_EWS was created as a new engine that builds on top of others. It will use a backing-store engine like X11, FB, DirectFB, WinCE or PS3 to create an internal Ecore_Evas. Every new window created with ecore_evas_ews_new() is rendered to an Evas using the buffer engine and its pixels end as the image source in the backing-store engine, displaying it. This is very similar to existing ecore_evas_buffer feature ecore_evas_object_image_new(), however it will handle more things automatically. Among the new features are events to allow window management. Things like "window created", "window moved" and "window resized" are placed in the main loop for interested peers, they may handle these and decorate the windows, offer window management (close, maximize, ...). To exemplify that I've added support in Elementary with basic controls such as move, maximize, restore and close. Unfortunately it looks ugly like hell and then I'm not posting any screenshot or screencast ;-) Let's wait for people doing themes to make a great work. The work is far from complete, bugs remain, optimizations could be applied for OpenGL... if you consider the rules "20/80", 80% of the code was written in 20% of the time, now we'll take 80% to make it work for sure :-D But it's not bad considering it took me 2 days, 1421 lines in ecore_evas_ews.c, 543 in elu_ews_wm.c and 317 in ews.edc... That's why I keep saying that EFL is an amazing technology to build products. It runs fast, it's slick and it does not get in the way. Give it a try! :-)0 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 06 Oct 2011 00:51:00 +0000 http://maemo.org/midcom-permalink-1e71f95afc2d1a01f9511e799c74d55c3ebdfbddfbd Introducing EWS to help EFL on FB and PS3 http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/5w8VA79XmM0/ Lots of systems do not support multi-window on their own, consider standard Linux Frame Buffer (FB) or the PlayStation3. This makes it cumbersome as you’re restricted to a single window,  likely you’ll have to rewrite your apps to behave well in this scenario.

Traditionally this problem is worked around by toolkits like Qt that provides QWS, a great help for developers. Considering QWS I decided to help our PS3 friends with something similar which I shamelessly called EWS (Ecore + Evas Single Process Windowing System).

Ecore and Evas already made 99% of this work: they abstract rendering, abstract windowing and so on. There is even support for “inlined windows” in Elementary, these are rendered to buffers that are visible inside other windows. Actually the first version of EWS was a 30min hack in Elementary called “eland” (making fun of wayland). But as not everyone would use Elementary it was better to move it lower in the stack: Ecore_Evas.

Ecore_Evas is a glue layer between Ecore and Evas that takes care to configure actual windows and setup Evas on them, to capture events from various sources and feed them to canvas, etc. If you cope with its API you’ll work with most EFL libraries and applications out-of-the-box.

Ecore_Evas_EWS was created as a new engine that builds on top of others. It will use a backing-store engine like X11, FB, DirectFB, WinCE or PS3 to create an internal Ecore_Evas. Every new window created with ecore_evas_ews_new() is rendered to an Evas using the buffer engine and its pixels end as the image source in the backing-store engine, displaying it. This is very similar to existing ecore_evas_buffer feature ecore_evas_object_image_new(), however it will handle more things automatically.

Among the new features are events to allow window management. Things like “window created”, “window moved” and “window resized” are placed in the main loop for interested peers, they may handle these and decorate the windows, offer window management (close, maximize, …). To exemplify that I’ve added support in Elementary with basic controls such as move, maximize, restore and close. Unfortunately it looks  ugly like hell and then I’m not posting any screenshot or screencast ;-) Let’s wait for people doing themes to make a great work.

The work is far from complete, bugs remain,  optimizations could be applied for OpenGL… if you consider the rules “20/80″, 80% of the code was written in 20% of the time, now we’ll take 80% to make it work for sure :-D  But it’s not bad considering it took me 2 days,  1421 lines in ecore_evas_ews.c, 543 in elu_ews_wm.c and 317 in ews.edc…

That’s why I keep saying that EFL is an amazing technology to build products. It runs fast, it’s slick and it does not get in the way. Give it a try! :-)

1 Add to favourites3 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 06 Oct 2011 00:51:00 +0000 http://maemo.org/midcom-permalink-54ac0f3aefbb11e0a31d4f86d58fcb68cb68 EFL portability shines: native PS3 support! http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/JFYDGFbwB7I/ While most projects praises portability with fancy things like “gchar” (Gtk), fancy names like “LightHouse” (Qt) or completely new abstraction concepts, The Enlightenment Project always choose a simple approach to use POSIX and fill the gaps as required. This requires much less work and yields the same or better results, as we could see in the past and now being the first toolkit to be ported to PlayStation3 (native, no Linux)!

As announced by the the developer, KaKaRoTo, in our official news the port was done on top of our existing SDL backend in a couple of days, then being migrated to native PlayStation3 using unofficial SDK PSL1GHT for jailbroken systems. There is still no hardware acceleration, yet it runs fine a physics game at Full HD.

Right now we have port to MacOS and Windows, including WinCE. Given our scarce resources we do not lag behind Qt or Gtk in portability front. Basically what we have is one brave developer called Vincent Torri that is a Math professor and do it on his spare time!

Vincent could do it by keeping our approach: use POSIX and fill the gaps. He created “Evil”, a library to provide missing functions to Windows. Kakaroto then created “Escape” to do the same for PSL1GHT. Seems this approach works fine ;-)

1 Add to favourites2 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Fri, 16 Sep 2011 19:22:11 +0000 http://maemo.org/midcom-permalink-b53dd732e09a11e0b75a13a7e6c00d450d45 DesktopSummit 2011 – Berlin http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/wKQMnB0vkHY/ I had the pleasure to attend the DesktopSummit 2011, a great event that happened in Berlin from 5-12 of August 2011. As I’ve mentioned in an earlier blogpost my focus was to highlight Enlightenment’s opinion that performance matters and that we need broader standards in freedesktop.org. Of course I explained a bit about our history and the current status of E17. The presentation file can be downloaded here.

All in all the event was great, not just due the talks but mostly due the friends and side-talks. Being at these events for a few years I managed to know lots of people from different projects. Not being in a major side (read: KDE or GNOME) I have the gift of free-transit among these fields… which is pleasant as I can gather ideas from both. [There is no hard or official barriers between them, but the psychological blocks peers from talking to each other and this is quite bad]

I came there sponsored by my company (ProFUSION embedded systems) to represent the system that provides a big bucket of our consulting, training and development services: Enlightenment.  I had the pleasure to engage into endless conversations with Enlightenment developers and community leaders (Cedric, Boris, Michael, Philippe…) We discussed a lot how we could broader EFL usage, bring more users, lower the barrier to new developers with easier to use tools and languages and of course how we could get Raster to release E17.

I also tried to learn from other people of technologies as well.  Before I was an Enlightenment hacker I did use and develop with Qt/KDE and already knew some icons such as Aaron Seigo, Thiago Macieira, Helio Castro and Sebastian Kügler, which I managed to meet again. Of course my ex-coworkers at INdT and KDE fanatics Artur Duque de Souza, Renato Chencarek were there. And I was introduced to Daker Fernandes Pinheiro, from INdT. We discussed QML, Qt, MeeGo and lots of optimization and API designs. Quite productive!

Being part of Maemo since 2006, attended some GUADEC and hacked Gtk/GNOME for fun and work I got to know some icons there as well. I’ve talked to Lennart Poettering, Marcel Holtmann and Marc-André Lueau, people that I’ve worked together in a way or another and that are always open minded to discussions. I’ve also touched base with some people like Lucas Rocha and Zeeshan Ali. It is interesting to know what these guys are doing for Linux Desktop (and mobile) infrastructure and their vision for GNOME.

My presentation went quite well, I was bit anxious and nervous in such a huge room that held it, but I guess people understood the history of Enlightenment, why we created the “Foundation Libraries” (instead of using Gtk or Qt),  our special care with performance and why it will always pay off. I did some heavy critics to FreeDesktop.Org that generated positive feedback from Thiago Macieira (Qt/KDE) and people from other desktops (XFCE/LXDE). Last but not least I’ve made it clear Enlightenment has serious problems to manage community, we’re quite bully, and did thank the guys like Philippe Caseiro and other french dudes that are trying to solve this issue.

All in all a great event, with great people! Looking forward to be in next desktopsummit as well!

5 Add to favourites1 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Tue, 23 Aug 2011 18:39:41 +0000 http://maemo.org/midcom-permalink-10847b3acdbf11e0ae2f6dc702f05f005f00 ANN: LightMediaScanner 0.4.4 released http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/XZfGlbP6YuU/ I’m proud to announce LightMediaScanner 0.4.4 was released and I’d like to take some time to remind you of this awesome project ;-)

While 0.4.4 is no big leap ahead of 0.4.3 I’m doing this post more to make clear people that the project is still alive and it may be the right tool for you, particularly if you need fast media scanning on your desktop or embedded device.

Introduction (History)

LightMediaScanner (LMS) was born in 2007 during the development of Canola2 media player for Nokia N800 Maemo device. We did a previous try at media scanners for Canola1 and did learn a lot, that would be the foundation of LMS.

We had two major problems to solve:

  1. scan large amounts of media as fast as possible
  2. don’t mess with interactivity.

The first problem had more qualifiers, like the media being unreliable (broken files), be removed without prior notification (mmc removed when system was off), slow discs (sd/mmc), small amount of available RAM, low-end CPU and so on.

Traditional approaches were slow and would bring your device to unusable state for minutes, even after the scan process finished as they consumed too much memory that would swap your applications and services. They would also be naive with file scanning and would trash filesystem’s cache with useless data.

Existing Solutions Problems

At that time we had Maemo’s media scanner that was awful and Tracker was being developed to solve its problems. However Tracker was immature and had other focus, more like being fancy with new standards and trying to solve all the diseases of the world… while we need to be fast in some specific cases.

To highlight technical problems, let’s consider JPEG scanners. Everything one Media Player application would need is the  title, author/artist, date, width, height, orientation and maybe geolocation. These are few bytes, particularly because Title and Author are missing in most cases. We should open the file, get these bytes, register these and close the file. Every system at the time used libexif to do so, however it would copy all the header to memory prior to scanning. Given that most files contain thumbnails in the header, it would mean useless malloc + read of around 10-50kb, to get dozen bytes and exit. This trashes the VM and disc. Moreover, some libraries would open the file themselves and would not fadvise as POSIX_FADV_DONTNEED, thus the filesystem would keep it in cache for further access that never happen! Similar situations existed for PNG, Mp3 and others.

LightMediaScanner Solution

We decided to go with a 2 process solution to avoid crashes and hang-ups on the main process. This would be safe and we could kill the slave process if it took too long to report on a given file. Later on we introduced threaded and single process scanning that would be useful in some cases.

We also had dynamic loadable plugins (shared objects) with the media parsers to be used given matching extensions. The plugins can be selected by the user at runtime. Yes, we would not scan every file with every parser, we did trust the extension as a way to save work and it worked fine, and fast.

The main process just scans the filesystem and  pipes the files to the slave (thread or process). This slave would open the file, get the data to populate a structure to be written to database, then close the file and report back the progress. If the slave takes too long to reply or dies, it’s respawned with the next file in the queue.

The database uses SQLite3 and a configurable number of files are processed in one SQL transaction, thus we’d not hit the filesystem with writes that often. The database access is provided with helper functions that uses pre-compiled statements. However we only provide the insert/update methods, as the fetch (select) are up to the application to use (maybe using their own ORM framework).

Some metadata formats did not specify encoding (ASCII was assumed) and fallbacks can be applied with the charsets. It will use iconv in a priority list to figure out what works.

Just the Indexer!

LightMediaScanner is not a daemon-service, it does not talk DBus neither requires you to speak  RDF or SPARQL. If you want, you can create one with it, but it does not provide this stuff out of the box. Basically you call it to check for media and then access the database yourself, using the SQLite3 directly or using some ORM, doing caches or not, it’s up to you.

The database tables are quite simple and try to provide common useful relations. It’s not designed to cover all weird cases in the world (like multiple artists per song or which lens did you use with your camera).

Using it!

Get the code from our GIT or use the release tarball, see http://people.profusion.mobi/cgit.cgi/lightmediascanner.git/. It uses standard autoconf/automake to build, so should be familiar to most developers.

It hard-depends only on SQLite3 and iconv (most libC ships with it). Optional dependencies are libvorbis, libmp4v2 and libflac.

You can call the scanner from C, C++ or Python using our bindings. Comprehensive example can be found at src/bin/test.c

References: previous post about lightmediascanner 0.2.0 release

8 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Fri, 19 Aug 2011 22:19:21 +0000 http://maemo.org/midcom-permalink-1c13ee22cab311e08e9f7ffc9723c247c247 Blog recovered http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/1XjHLdlqKEY/ Hi all,

After a while not even opening my blog, yesterday I did two posts. While the administrator interface seemed fine, readers quickly notified that it was showing lots of spam in the regular view which I confirmed using Chromium’s private browsing. Investigations led to dozen administrator accounts in WordPress database, then I decided to reinstall from scratch. Unfortunately yesterday was a busy day and I could barely stay at the computer to do so.

Anyway, this morning I restored my blog and I’ll try to keep it updated :-D I also changed the comments rule, instead of requiring people to register, I instead opted to close comments after 14 days, since most spammers seems to look for pages with reasonable pagerank and new pages do not have them that soon.

3 Add to favourites5 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Fri, 03 Jun 2011 16:32:29 +0000 http://maemo.org/midcom-permalink-9571f8188dff11e08d95f5b11686ed40ed40 ESC Brazil – RealTime Linux with RT_PREEMPT http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/wk0nmUiBMP4/ The first Embedded Systems Conference Brazil was held at São Paulo on 24 to May 25 2011 and ProFUSION was there to do a technical talk.

Our contributor Lucas De Marchi proposed the talk “Usando Linux como Sistema de Tempo Real” (Using Linux as an Real Time Operating System). He did his master thesis on that topic for 2 years at Politecnico di Milano, so he knows a lot about the topic.

Unfortunately Lucas is one of those geeks that try to live a real-world life and plays football (soccer for americans) and in the week before he broke his foot… Ouch!

So yours truly was elected to present the talk for him. I already did play with realtime Linux in the past and I knew the concepts from past experiences, thus I accepted the challenge and did the talk. According to the attendees it was a good talk!

People got amazed with Linux’s Real Time capabilities and the ability to mix it with regular (“fair”) tasks, being able to change a process from FAIR to RT during runtime, no recompile was needed! This was demonstrated in a Freescale’s i.MX31 running music player running with regular priority and ping flood and a heavy CPU task, the music skipped and it was clear the deadlines were not being met. Running chrt and raising the priority above Kernel Threads that handle interruptions immediately fixed the skips. We also ran a software oscilloscope at desktop that plotted the achieved deadlines and it confirmed what we were listening.

If you’re not aware, RT_PREEMPT is a patchset (from pre-git days, not a git tree called -rt) that Ingo Molnar and Thomas Gleixner run to improve Real-Time behavior for the Linux kernel. As concepts are tested and patches mature they are moved into mainline tree, benefiting everyone using it, even your desktop running PulseAudio now! The major trick of this work was to handle interruptions in Kernel Threads instead of immediately as it used to be. These threads are RT themselves at priority 50 and if you want you can have a regular user process thread at a higher priority and it will run before the kernel. This was the case with the pingflood, with a regular kernel the music could skip due kernel stopping the playback process to handle the network interruption.

Last but not least, if you’re not familiar with RT have something clear in your mind. Real Time doesn’t mean Real Fast. It just mean you have guarantees to execute things when they should and this is mainly done by means of a preemptive kernel. In some cases RT will turn things slower, as you may switch contexts more often due preemption, but you’ll know you’ll be on time (as opposed to batch processing). Linux being a General Purpose Operating System (GPOS) was not designed with such thing in mind, but it was successfully converted into one!

3 Add to favourites1 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 02 Jun 2011 16:03:50 +0000 http://maemo.org/midcom-permalink-da69c6508d3511e0ad8325319c07b823b823 DesktopSummit 2011 http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/9ZcOPJxssk8/ Hi all,

After I started on twitter and facebook the blog posts reduced a lot… anyways I’d like to mention I’ll attend DesktopSummit 2011 in Berlin to represent the Enlightenment (aka E17) team and thus an alternative view to the traditional GNOME x KDE.

Fortunately I have couple of friends in both GNOME and KDE so I can express our E17 voice without being ignored (at least immediately ;-P). Dunno if it’s me or E17 being the underdog, but people don’t feel afraid to listen to our ideas and they even try to incorporate those that make sense, like Enlightenment’s Edje that resulted in QEdje that in turn resulted into QML that boosts Qt development these days.

My focus will be:

  • performance matters: even if you have a fast hardware today and an even faster hardware tomorrow, you should care about performance. I’m not talking about neat picking premature optimizations, but to design platforms and standards considering performance. It’s often not harder, just need to be considered from the beginning. What bugs us most is the stupid freedesktop.org thumbnail standard, that forces use of PNG even if most of our pictures are in JPEG, causing slower thumbnailing time, bigger disk footprint and slower load times. Second in the list is the traditional abuse of XML, praised years ago and suddenly GNOME realized it was a bad idea (given GSetting/DConf), but it’s deeply inside our desktops with fontconfig and others (want faster application load times? Don’t use XML.);
  • Broader Standards: A standard must be thought outside the “I’m implementing it for myself” scope. Similar to above, we should consider broader standards and not get a piece of GNOME or KDE and say it’s a standard. Freedesktop.org fails to get standards because of this stupid approach. See the last flamewars around simple topics as the systray replacement. Actually the systray standard is a well known bad example, people wrote it like if GNOME was the only way to do this. Enlightenment officially ignored this standard due stupid bugs with XEmbed specification and approach (as EFL uses the “windowless widgets” for a while, the XEmbed made no sense and caused user experience problems);
  • Embedded is the future. Okay, it will be a DesktopSummit, but as Rasterman predicted  in 2002 the desktop game is unlike to be changed from Microsoft dominance and we better look for new fields where we can compete equally. Given Android, WebOS, Maemo, MeeGo and others, it’s clear that Linux does dominated the mobile world. With appliances (from TVs to DVR to Refrigerators) being dominated for a while (after the embedded cpus got more powerful, most embedded OS got replaced with Linux). Thus together with performance we must not restrict ourselves to the keyboard-mouse-X11-big-screens approach that we used before. This is not just about scalable (as in multiplication) GUI as people believed SVG would help, but we need different GUI for different environments and our toolkits and infrastructure have to help with that… something that Apple is currently doing quite well with their iOS UI toolkit providing universal applications that run optimized on both iPad and iPhone/iPod.

If you’re also attending this summit, let’s meet there and have some beers.

3 Add to favourites2 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Thu, 02 Jun 2011 15:32:37 +0000 http://maemo.org/midcom-permalink-d99a41be8d3511e0ad8325319c07b823b823 ANN: liblogger http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/5WaOgRKCIPA/

“There is no better tool to debug than printf()”
– Latin proverb from an old Chinese man.

That’s what every programmer knows. Despite interesting tools like profilers, debuggers and other tracing tool, for a reason or another we often resort to good old printf(). The logic behind this is quite simple: it’s available everywhere and have no impact on performace (mostly, on both cases — don’t be picky!), yet it allows you to check out if your program flow is as you expected and if not, what’s happening. The bad part, of course, is that it’s invasive and cumbersome to type, often introduces errors by itself (come on, raise your hand if you never ever wasted hours debugging to notice that the bug was in the debug message!)

GDB and some tools like ltrace or GLibC’s latrace are not always available, particularly on embededd systems they may be available but almost unusable. I hit that in the past weeks during some debugging on an old MIPS where I had to investigate why my supposedly correct program did not work, but a proprietary one did. Documentation is almost inexistent and makes you more clueless than not, thus I had to compare what we were doing differently.

At first I started doing an old trick to create a shared library to be $LD_PRELOADed on my own that would log what was being called, then dlopen()/dlsym() the actual symbol, call it, then log again, then return. I did it for a handful functions, some I even printed out structure members and although it worked, soon I got bored and hit by the laziness that most programmers have: I must make this automatic. Liblogger was born.

Liblogger is Python program that will parse a C header file and generate a source file that does exactly what I described above: log entrance, call the actual function and keep its return value, then log exit, then return the value. That’s the basics this single-file without external dependencies do. This source file can then be compiled into a shared object or into the final application.

Of course I wanted it to be as helpful as possible, so this first version already includes some nifty features:

  • source file can be compile-time configured using CPP defines to produce colored, thread-safe, timestamped (gettimeofday, clock_gettime), indented (nesting level) output. This output can go to stderr or some given file;
  • generates makefile for you, creating most of aforementioned variations by default. Just pick the one you want to use and LD_PRELOAD it;
  • generate types file with all parsed enums, structs, unions and typedefs;
  • generate formatters file with custom functions to print out known enums, structs and unions. No more copy + sed on the original structs to produce printf() statements manually!
  • will keep errno intact so functions that rely on it will work as it will not be reset by fprintf() calls;
  • can be compiled in a thread-safe way, doing locks around dlopen(), dlsym() and the log FILE* to keep messages atomic. It will also keep the indentation level per-thread and print out thread ids (if not main thread);
  • Behavior is completely controlled with a INI-style configuration file. It can (check for more at README.txt):
    • tell the parser tokens to ignore;
    • filter (positively/negatively) functions to log;
    • specify type formatters you want to use. Couple are provided for basic C types, and you can request it to generate custom formatters for parsed types as mentioned above;
    • specify return value checkers you want to use. Couple are provided, including one that checks for errno and logs the value as a string;
    • if types of parameters and return are safe to be used by formatters, this can be used at various levels to provide black-white lists for these and keep it useful and safe;
    • special handling of return (output) parameters, their values can be dereferenced and printed out after the actual function is called;

Usage should be pretty straightforward but I distribute two examples, one for dbus and another for jpeglib. Part of the dbus example is reproduced below:

  • dbus.cfg
    • [global]
headers = dbus/dbus.h
overrides = formatters-%(header_name)s.c
# regular expression for all defines in dbus-macros.h
ignore-tokens-regexp = (DBUS_MACROS_H|DBUS_BEGIN_DECLS|DBUS_END_DECLS|DBUS_BEGIN_DECLS|DBUS_END_DECLS|DBUS_DEPRECATED|DBUS_DEPRECATED|DBUS_DEPRECATED|DBUS_EXPORT|DBUS_EXPORT|DBUS_EXPORT|DBUS_EXPORT)

[type-formatters]
DBusHandleMessageFunction = %(prefix)s_log_fmt_pointer
DBusFreeFunction = %(prefix)s_log_fmt_pointer
DBusHandleMessageFunction = %(prefix)s_log_fmt_pointer

[safe-formatters]
DBusHandleMessageFunction = true
DBusFreeFunction = true
DBusHandleMessageFunction = true

# be pedantic about safety, let's specify all strings independently

[func-dbus_message_new_method_call]
parameter-bus_name-safe = true
parameter-path-safe = true
parameter-interface-safe = true
parameter-method-safe = true
  • call: 
    liblogger.py \
    --config dbus.cfg \
    --makefile Makefile.dbus-connection \
    --makefile-cflags "\`pkg-config --cflags dbus-1\` -I." \
    --custom-formatters formatters-connection.c \
    /usr/include/dbus-1.0/dbus/dbus-connection.h \
    libdbus-1.so \
    log-dbus-connection.c
make -f Makefile.dbus-connection
LD_PRELOAD=./log-dbus-connection-color-timestamp.so d-feet
9 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Wed, 09 Mar 2011 21:38:29 +0000 http://maemo.org/midcom-permalink-698e868a4a9d11e09b469957d46f11651165 MeeGo, Nokia… light at the end? http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/B1VukszaeQI/ Hello people, as most of open source lovers I’m still trying to digest Nokia’s move to WP7. But although many are worried about the future of Meego or the Linux on mobiles, I’m quite confident.

First, as said before MeeGo is not just about Nokia or phones, it’s also being pushed as In-Vehicle Infotainment (IVI) and is already adopted by BMW, which you can see daily contributions on projects such as ConnMan. So let’s not assume it is the end of the world given Nokia’s action.

Second, although Android and WebOS are indeed Linux since they use this kernel, we often want more traditional user-space stack, like MeeGo would be. For those,  don’t loose hope! Today (Feb14, 2011) we got a major announcement at MWC by LiMo: LiMo Foundation Unveils LiMo 4. It’s based on X11, WebKit, GNOME and… EFL!

Wait? Where do you see EFL in that announcement? You’ll have to check What is the LiMo 4 Platform? block diagram and see EFL is now a first class component at it, together with GNOME that was there since first release. Yeah, we surely could use better publicity at EFL side ;-)

The release is still just text and no code, which should be available by July2011, however you can get code straight from EFL SVN.

ProFUSION team is happy due our contributions to make this possible, after all we help with EFL and WebKit-EFL developments!

8 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Mon, 14 Feb 2011 20:53:28 +0000 http://maemo.org/midcom-permalink-25c15fae388211e083de43184d653eb33eb3 systemd! http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/fzTxv_k6AeM/ Yeah, I did not write for a while, but that does not mean I’m dead or changed interests, just that running an always growing company is taking lots of time, with the spare time I get being invested in something interesting and relevant, usually that does not translate to blog posts :-P  However I felt like doing it today.

While most people know me due my graphical user interfaces work, likely related to Enlightenment, I’m one of those guys that like to consider the system as a whole and not just a subset of it. My main interest since 1998 is graphics, from good old days of tcl/tk and gtk1 to pygame/sdl to qt3/4 and later enlightenment. But I always kept an eye on other bits like boot, networking, sound and so on.

After getting into ConnMan to have a good (and fast) network infrastructure, I started to consider similar bloat-reducing options to other parts of the system. Talking to friends like Holtmann, Lennart and ProFUSION co-workers about the stupid stuff we do in our distributions, like abusing shell scripts for no good… even tried to resurrect Gentoo’s bug #69579 (Bootup Time Reduction/General Speedup).

My first idea was simple, hackish and selfish: as I’m Gentoo user, I could write a shell script compiler that generates threaded-C code with the exact state machine I’d have using /etc/init.d with OpenRC. It was simple (or not so complex) as most of the scripts are start-stop-daemon wrappers with dead-simple usage of OpenRC’s dependency information (need, use, after, before…). However it was hackish because couple of them are not simple and would require hacks, either special cases or rewrite them natively, to make it work. Last but not least, it was selfish as likely it would just fix my problem, or with some luck it could benefit Gentoo as a whole. All of that after I loosing faith in Upstart of couple of tries and talks with related developers.

However sometimes it is better to talk to smart(er?) people before you go do your own stuff. I don’t feel like I suffer from NIH-syndrome. So talking here and there, João Paulo pointed that Lennart had a still secret project to do exactly what I was aiming, but knowing Lenny, it ought to be good. So I immediately /q -freenode mezcalero and started chatting, with he demonstrating a great cleverness and even proving me wrong in some assumptions, he already thought of much more like merging inted features and how to isolate services in their own control groups… yeah, he got me in systemd’s bandwagon.

As I still have tons of other stuff to do, I put the idea on hold and went back to it after several posts and announcements that Fedora 14 would use it. At the first free time slot I checked out the code and Gentoo’s “port”. Well, as expected with young projects it did not cover some special cases, like systems without IPv6 (mine) and Gentoo’s /etc/init.d scripts were too smart due OpenRC’s dependency resolver that would make their usage more harm than good. So I had work to do, but it was done and contributed back, with awesome results.

The project and community, also composed of other nice and helpful hackers such as Kay Sievers and Michael Biebl and I felt like it was an excellent project ProFUSION could collaborate, then we invested 2 developers to help it: Fabiano Fidencio and Lucas de Marchi, that got it running on Arch (OpenEmbedded on the way) and added some missing features, like native poweroff/halt/reboot, compile time toggle to disable SysV backward compatibility and so on, runtime fallbacks if no IPv6 and so on.

Right now systemd is very powerful, it can almost fulfill all my needs and likely can manage everything an embedded system would use on its own. From its lightness that reduced my boot in 10s and 2000 process, to nice helpers like its syslog-kmsg bridge. In my opinion, the biggest missing feature is cron-like features to reduce yet-another daemon from boot and centralize it in the “system daemon — systemd”.

If you don’t know it yet, give it a try. It’s young however very mature, handling complex cases, providing nice utilities and trying to unify Linux boot process. A short list of its awesomeness would be:

  • each service is isolated in its own control group, being able to apply limits and avoid children processes to exit their parent’s control (ie: php/perl from cgi-bin exit from their apache tree). Aside from safety, this allows monitoring when a forking service died as all process in its cgroup are gone.
  • socket activation, like inted/xinetd, allows seldom used servers to not add load to boot process yet they still work as expected, like sshd and cupsd for desktop systems.
  • lazy mount of filesystems by means of kernel automount. Similarly to socket activation, this lazy mount avoids adding load to boot process and postpone the actual mount when the system actually need files from such mountpoint.
  • easy to read/write configuration files (units) with ini-like syntax, matching other freedesktop.org projects such as dbus.
  • native support, properly written in C, to various boot tasks such as loading modules, load/save random seed, clean temporary files, setup console fonts and keys, enforcing quota checks, creating required directories with correct permissions and so on.
  • useful helpers such as systemd-kmsg-syslogd that directs /dev/log to /dev/kmsg, that avoids the need of a proper system logger with logrotate on embedded (or desktops that doesn’t need persistent logging).
  • awesome readahead code that uses fanotify (temporarily disabled in 2.6.36) to passively collect read files, even while replaying already collected from previous boot — without stepping on its own toe or major performance hit.
  • proper integration with DBus to provide race-free activation of system services.
  • various agent interfaces to request user intervention, like passwords.
  • proper remote control interface (uses DBus)
  • proper (optional) SELinux and auditd labeling and interaction.

All in all I really like Systemd and I foresee a brilliant future for it, given major distros adopt it officially (Fedora and OpenSuse at least?)

9 Add to favourites0 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Fri, 29 Oct 2010 22:41:50 +0000 http://maemo.org/midcom-permalink-03ba3ac4e3b511dfa3db69800ebfc038c038 United “sucker” Airlines http://feedproxy.google.com/~r/GustavoSverzutBarbieri/~3/h3lhCLMRkPY/ Notice: this is a rant posted only to my “life” blog category, if you happen to not like personal stories but got this due some syndication using my global RSS instead of specific, please forgive me and just ignore this post.

Ulisses and I had to travel to a conference in Orlando, FL and we did the stupid mistake of booking the flight with United Airlines. Yes, we already had problems with it in the past and we knew United Breaks Guitars, but nonetheless we insisted to give it another try. It was already strange that our flight to Orlando had to go to Washington DC in the first step, then we would have to go through Chicago during our return… but yeah, it was cheaper and the airlines do these stupid routes to aggregate more people.

Problem#1: So… Sunday, June 20th, we were set to depart at 10pm. Given that my birthday was the day before, I was at my family’s house for a great Brazilian BBQ. However their city is 4 hours away from the airport, you all know the airport checks take some time and they require us to be 3 hours before, so 7 hours earlier I departed. If you happen to like football as I do, and you are Brazilian, you would rather die instead of loosing Brazil versus Ivory Coast in the world cup, but I was forced to.

Arriving at the airport exactly 3 hours earlier, we did the check-in, went through security and waited for hours… to be notified 20 minutes before that our flight was cancelled due no reason. They rumored that the aircraft had mechanical problems and could not fly.

Problem#2: We were told that we should go in the next flight at around 1pm the next day… it was already bad, given that we’d loose the first conference day, but it became worse once other passengers alerted us that UA actually did not had any flights departing at that time! Apparently they were just doing that so they would just have to pay us one hotel night, forcing us to be at the airport in the morning and then wait until 10pm for the actual flight!

We could not afford such delay as the second conference day was the one with the important bits. Thus we made them book us a flight with their partner company “TAM”, departing Monday 21th in the morning. The flight was a direct one and that was awesome, with a new airplane with interactive media displays and so on.

Problem#3: We had to go to Chicago where we had the connection back to São Paulo, it was around 2 hour slack between them and so good balance, not much to wait and enough time to get our luggage transferred. But UA was nice to delay our flight departure by 30 minutes. A bit worried, we were confident it was still enough. However during the flight, the pilot announced “We’ll have to stop in Kansas to refuel”, WTF we thought immediately. Of course this was a stupid excuse for something that they did not tell us. Kansas it was, long delay… but we did manage to get to Chicago 30 minutes before our flight departed.

Obviously, while you can walk from one gate to another in 30 minutes, your luggage cannot go through their unknown process and it did not make into the plane.

Problem#4: We arrived in São Paulo and waited for our baggage… thinking in a logical way, we did know it was not there, but as the government slogan says “we’re brazilian and we’ll never loose hope!” After confirming that it was not there, we went to United Airline help desk. Strange looks, computer queries and questions about our route in USA, they said it was located in Chicago and should be in Brazil in their next flight, which should be this afternoon (June 29th). The whole they happened, but no news from our luggage… called them and they said it is still in Chicago, but at this point: would you believe them?

Problem#5: when entering Brazil you have the usual options “Nothing to declare” or “Goods to declare”. Often then choose some samples from “nothing to declare” to be exhaustively investigated, a quite invasive search I’d say. We did not had anything to declare, but given that our luggages were not there, we had to go be searched just to be able to have our stuff to be allowed through the Customs without us. WTF, what is the logic behind that?! A huge inconvenience.

Problem#6: my luggage was package with all my brand new clothes as we had scheduled some meetings with costumers in the conference. For some stupid mistake, I also dispatched my BlackBerry with my luggage since I did not use it for the whole week. Ulisses also had some nice gifts for his daughter in his bag. Very, very likely we’ll not see such personal items anymore, as they are likely to vanish with our luggage or just handle us opened bags without these items as it already happened to me in the past.

Yeah, United breaks guitars, make you loose your football game, fuck with your business travel, loose your luggage… what a shame, never using United services in future, this time for real. I did learn my lesson.

5 Add to favourites3 Bury]]> Gustavo Barbieri <barbieri@gmail.com> feed:01c9bb92c8cdc7e3a5fc627f123acc22 Tue, 29 Jun 2010 23:11:52 +0000 http://maemo.org/midcom-permalink-66cd77fe83d711df80d99f47cadce60be60b