chinook

Midgard Administrator — Tue, 24 Jun 2008 11:14:35 +0000

The SDK tools documentation for maemo 4 Chinook is available from the chinook tools page:

http://maemo.org/development/tools/chinook.html

sp-endurance

Juha Kallioinen — Wed, 09 Apr 2008 10:07:01 +0000

Description

sp-endurance provides tools to save various system endurance related information (resource usage, errors) on device.

Provided tools include:

binaries for saving /proc and X client resource usage as CSV
a script for saving these along with syslog, SMAPS data, df and ifconfig output etc.
a script for getting quick system memory usage change overview from the endurance data

The endurance data can be used for detecting and measuring resource leakage and application errors.

Packages

source: sp-endurance

binary: sp-endurance

Usage

Because resource leakage detection requires repeating the given use-cases many times, it's best to automate the running of the test-cases. Both before the first and after each additional test-case run the following can be done:

# save-incremental-endurance-stats use-case-name
# cat /dev/null > /var/log/syslog
# cat /dev/null > /var/log/syslog.old

This will create a directory called use-case-name (if it doesn't exist) and add on each invocation a new subdirectory with the current endurance data. The subdirectories are named 101, 102 and so on.

Syslogs can be large and they are included into the endurance data so it's best to clear syslog(s) (as indicated above) after the endurance data is stored. Otherwise storing the data can take quite a long time and potentially consume huge amounts of disk space.

Note: If you run this in the device over ssh instead of from console, you might not have DISPLAY set correctly and then xmeminfo complains that it cannot connect to X server. In that case, just enter:

# export DISPLAY=:0

before starting your testing.

Emphasizing leakage

Sometimes it is not clear from the endurance data whether there's a memory leak or not. Increasing the number of test-case runs helps, but getting more measurements doesn't necessarily make it easier to see the leakage. In this case it is best to emphasize the leakage by increasing the number of test-case repeats between measurements exponentially, like this:

#!/bin/sh
measurements=5
repeats=1
for round in $(seq $measurements); do
        echo "round $round:"
        # repeat the test-case round^2 times
        for repeat in $(seq $repeats); do
                echo -n " $repeat"
                # RUN THE TEST-CASE HERE
        done
        repeats=$(($repeats+$repeats))
        save-incremental-endurance-stats use-case-name
        cat /dev/null >/var/log/syslog
        cat /dev/null >/var/log/syslog.old
done

Post-processing

After the tests have been completed, the stored endurance data can then be transferred to PC for post-processing with sp-endurance-postproc.

Note that the post-processing tools create the error and resource leakage summary from the differences between 2nd and last measurement. They assume that:

the first measurement is "initial state" i.e. the system and application are still in a state where things needed by the test-case are not yet initialized
processes involved in the test-case haven't been terminated before the last measurement. If the process doesn't exist anymore in the last measurement, there's nothing to differentiate

On the device a rough system memory overview of the endurance data can be obtained with:

# endurance-mem-overview use-case-name/*

Links

Man pages:

endurance-mem-overview

proc2csv

save-incremental-endurance-stats

xmeminfo

oprofile

Juha Kallioinen — Wed, 09 Apr 2008 10:02:04 +0000

Description

Oprofile is a low overhead system-wide profiler for linux. It can be used to find CPU usage bottlenecks in the whole system and within processes.

Packages

source: oprofile

binary: oprofile, oprofile-common, oprofile-common-dbg

Installing Oprofile

Configuring the device

In order to run oprofile on your device, the device's kernel needs to be oprofile-enabled. A pre-compiled kernel with oprofile support is provided in the rx-34-kernel-oprofile package.

Getting a new kernel

The kernel package can be downloaded from the Chinook SDK repository in scratchbox environment (in an ARM target) with the command

(sbox-arm: ~) > fakeroot apt-get install rx-34-kernel-oprofile

This will install the kernel image to /usr/share/osso/kernels/

(sbox-arm: ~) > ls -l /usr/share/osso/kernels/
-rw-r--r--  1 user users 1536760 Oct 24 11:13 zImage-oprofile-rx-34-200743

the numbers at the end of the filename resemble a date and will be different if the kernel has been updated since the writing of this document.

By copying the image to outside of the scratchbox chroot environment, you'll be able to easily access it when you're ready to flash the image.

(sbox-arm: ~) > cp /usr/share/osso/kernels/zImage-oprofile-rx-34-200743 /tmp

Using the new kernel

There are two ways to use the kernel image. You can flash it to the device or you can just boot the device so that it uses this kernel until you reboot it again.

Option 1: Flashing the kernel

To flash your new kernel image, use the linux flasher utility (as root)
```
$ flasher -f --kernel /tmp/zImage-oprofile-rx-34-200743 -R
```
Flashing just the kernel does not destroy your other data. After the flasher has finished, your device is ready for oprofile.
Option 2: Boot with the new kernel
```
$ flasher --load --boot --kernel /tmp/zImage-oprofile-rx-34-200743
```
With the boot option the power cord has to be kept plugged in until you want to revert back to the previous kernel.

Restoring the normal kernel

After you are done with oprofile you can restore the normal kernel. How you do this depends on which one of the options above you used to load the oprofile kernel.

Option 1: You chose to flash the oprofile kernel

In this case the normal kernel needs to be re-flashed:
```
$ flasher -f --flash-only kernel -F <FIASCO image> -R


# The FIASCO image is the whole product image with a name like:
# RX-44_2008SE_1.2007.42-18_PR_COMBINED_MR0_ARM.bin
```
Note that just re-flashing the kernel part does not overwrite any of the other parts, so your data and settings will be intact.
Option 2: You chose to just boot with the new kernel

Restoring the old kernel is as easy as unplugging the power cable and power cycling the device.

Recognizing a suitable kernel

It's easy to see if your current kernel does not support oprofile by testing it with opcontrol:

Nokia-N810:~# opcontrol --status
modprobe: cannot parse modules.dep
modprobe: cannot parse modules.dep
Kernel doesn't support oprofile

Installing oprofile to the device

Providing that you have the Chinook SDK repository in your APT sources.list, the easiest way to install oprofile is using apt.

Nokia-N810:~# apt-get install oprofile

This will also install binutils, libpopt and oprofile-common.

Installing debug symbols

In order to view any useful profiling information at functions level, you will have to install debugging symbols. Debugging symbols normally come with debugging (-dbg) packages. The easiest way to install all dbg packages required for a given binary is to use debug-dep-install script which comes with the maemo-debug-scripts package:

Nokia-N810:~# apt-get install maemo-debug-scripts
Nokia-N810:~# debug-dep-install /usr/bin/osso-xterm.launch

Usage

On the device, type:
```
Nokia-N810:~# opcontrol --init
Nokia-N810:~# opcontrol --no-vmlinux
Nokia-N810:~# opcontrol -e=CPU_CYCLES:100000
```
100000 indicates the number of CPU cycles between interrupts. Increasing this number lowers down the accuracy but decreases the CPU overhead. --no-vmlinux indicates we are not interested in the kernel or do not have an unstripped kernel image.
Start the usecase you are interested in and type:
```
Nokia-N810:~# opcontrol --start
```
When you've finished, type:
```
Nokia-N810:~# opcontrol --stop
```

Now you've collected the data.

Viewing profile reports

To see basic per-process picture, type opreport:

Nokia-N810:~# opreport
CPU: ARM11 PMU, speed 0 MHz (estimated)
Counted CPU_CYCLES events (clock cycles counter) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CYCLES:100000|
  samples|      %|
------------------
     1677 69.1546 no-vmlinux
      240  9.8969 libc-2.5.so
      230  9.4845 busybox
      215  8.8660 ld-2.5.so
       58  2.3918 oprofiled
    3  0.1237 ophelp
    2  0.0825 libcrypt-2.5.so

To see more detailed symbol analysis use opreport -l:

Nokia-N810:~# opreport -l|more
warning: /no-vmlinux could not be found.
BFD: /usr/lib/debug/lib/ld-2.5.so: warning: sh_link not set for section `.ARM.exidx'
BFD: /usr/lib/debug/lib/libc-2.5.so: warning: sh_link not set for section `.ARM.exidx'
CPU: ARM11 PMU, speed 0 MHz (estimated)
Counted CPU_CYCLES events (clock cycles counter) with a unit mask of 0x00 (No unit mask) count 100000
samples  %        app name                 symbol name
1695     65.3179  no-vmlinux               (no symbols)
255       9.8266  busybox                  (no symbols)
222       8.5549  oprofiled                (no symbols)
43        1.6570  libc-2.5.so              strchr
37        1.4258  ld-2.5.so                check_match.0
32        1.2331  ld-2.5.so                do_lookup_x
17        0.6551  ld-2.5.so                _dl_relocate_object
17        0.6551  libc-2.5.so              _dl_addr
16        0.6166  ld-2.5.so                strcmp
14        0.5395  ld-2.5.so                _dl_lookup_symbol_x
13        0.5010  ld-2.5.so                __udivsi3
13        0.5010  ld-2.5.so                _dl_fixup
12        0.4624  libc-2.5.so              _int_malloc
10        0.3854  libc-2.5.so              strcmp
8         0.3083  ophelp                   (no symbols)
7         0.2697  libc-2.5.so              strpbrk
7         0.2697  libc-2.5.so              vfprintf

Profiling with callgraphs

Quite often a basic flat profile is useless. In such a cases a callgraph profile can be used. In order to profile code with callgraphs:

Add -fno-omit-frame-pointer to GCC options and recompile all the code (binaries, libraries) involved. You can do this without changing the package build rules by setting the appropriate scratchbox environment variable (documented in the Scratchbox installation /scratchbox/doc/variables.txt file) before re-building the packages:

(sbox-arm: ~) > export SBOX_BLOCK_COMPILER_ARGS="-fomit-frame-pointer"
(sbox-arm: ~) > export SBOX_EXTRA_COMPILER_ARGS="-fno-omit-frame-pointer"
(sbox-arm: ~) > cd package-1/
(sbox-arm: ~/package-1) > dpkg-buildpackage -rfakeroot
(sbox-arm: ~/package-1) > cd ../package-2/
(sbox-arm: ~/package-2) > dpkg-buildpackage -rfakeroot
...

Install re-built code and debug packages on the device

Init oprofile as usually, but add -c parameter:

Nokia-N810:~# opcontrol --init
Nokia-N810:~# opcontrol --no-vmlinux
Nokia-N810:~# opcontrol -e=CPU_CYCLES:100000 -c

Add -c to opreport:
```
Nokia-N810:~# opreport -l -c
```

Viewing reports from a PC

opreport -l, and especially opreport -c -l can take quite a long time (10 minutes) when fired up on N800/N810 devices. Therefore, it often makes sense to run opreport in scratchbox.

Configure scratchbox target in a way that its binaries and libraries 100% match the target's.
Collect profiling data as usual
Copy contents of /var/lib/oprofile from the device to the corresponding directory in scratchbox target.
in scratchbox, apt-get install maemo-debug-scripts (this may not be omitted)
install debug packages either with debug-dep-install or by hand

Note: the binaries and libraries in the scratchbox target must match what's in the device, otherwise you will get bogus results.

Oprofile with kcachegrind

kcachegrind is a useful GUI tool for viewing performance data interactively. It comes with many modern linux distros.

To use it:

Get the callgraph oprofile data (see above) and install the same packages also to scratchbox.
Copy the profile data to scratchbox session as described above.
install kcachegrind-converters package on HOST (debian, ubuntu)
in scratchbox: opreport -gdf | op2calltree (you might want to copy op2calltree script somewhere on target)
the resulting files can now be opened with kcachegrind on host, provided you set it to display ALL files (extensions are wrong)

Links

oprofile man page

traceroute

Juha Kallioinen — Wed, 09 Apr 2008 10:01:22 +0000

Description

Traceroute prints the route packets take to a network host. It displays each network hop and the time taken to reach it.

Traceroute is similar to tracepath, but offers more options and with some options requires superuser priviledges to run.

Packages

source: traceroute

binary: traceroute

Usage Examples

Find out the route the packets take from your machine to www.maemo.org:

# traceroute www.maemo.org

Links

traceroute man page

netcat

Juha Kallioinen — Wed, 09 Apr 2008 10:00:34 +0000

Description

Netcat is a simple unix utility which reads and writes data across network connections, using TCP or UDP protocol.

Packages

source: netcat

binary: netcat

Usage Examples

The utility is actually named nc. Here's an example of how to open up a TCP listening socket on port 5060. If someone connects to and sends data to that port, it will be printed on the screen.

# nc -l -p 5060

Links

netcat man page

iputils

Juha Kallioinen — Wed, 09 Apr 2008 10:00:07 +0000

Description

iputils is a small collection of TCP/IP related utilities.

ping: is already provided on the device by BusyBox and it's a tool to monitor and test network functionality.
arping: sends ARP requests to a host.
tracepath: trace a path to a host discovering MTU along the way. This is similar to traceroute, but does not require superuser rights or offer fancy options.

Packages

source: iputils

binary: iputils-ping, iputils-arping, iputils-tracepath

Usage Examples

To test if a host is in the network and responding to ping:

# ping host.name.net

Send four ARP requests to host:

# arping -c 4 host.name.net

Discover the path to www.maemo.org:

# tracepath www.maemo.org

Links

arping man page

tracepath man page

xresponse

Juha Kallioinen — Wed, 09 Apr 2008 09:59:22 +0000

Description

Xresponse synthetizes X stylus events and outputs screen update sizes and their timestamps.

Xresponse can catch applications doing screen updates which are not done as a response to user input/request and updates of unnecessary parts of application UIs. These are detrimental to application performance and device use-time.

Xresponse can also be used to time the applications screen updates i.e. their response time.

The screen update areas are output in the "X geometry" format (WIDTH x HEIGHT + X + Y).

Packages

source: xresponse

binary: xresponse

Usage Examples

Monitoring screen updates

Xresponse can be used to monitor how large parts of the screen and how often an application is updating them. To run it in the screen update monitoring mode, use:

# xresponse -w 0 -i

Then just use the applications.

Timing application response / screen updates

Xresponse can synthetize stylus taps, just specify where they should happen with the -c option and how long Xresponse should wait for the screen updates before proceeding with the -w option:

# xresponse -w 5 -c 200x20

If you want to monitor only part of the screen, e.g. toolbar, you can give which area of the screen is monitored with the -m option:

# xresponse -m 800x80+0+400 -w 20 -c 650x430

(This is useful for example if an application like Browser can be updating the normal application area also without user interaction.)

To time application startup, open the Task Navigator application menu, synthetize a tap to the menu item starting the application and monitor the screen updates. With most applications the last screen update is the time when the application is fully initialized and ready for use.

Links

xresponse man page

nano

Juha Kallioinen — Wed, 09 Apr 2008 09:58:21 +0000

Description

nano is a small and friendly editor that also supports stylus clicks to place the cursor and to execute shortcuts.

nano-tiny is a special minimal build of nano aimed for boot-floppies or emergency disks.

Packages

source: nano

binary: nano, nano-tiny

Usage Examples

Open up a file for viewing and editing. You can move around the file with the cursor keys.

# nano myfile.txt

Exit from nano: Press control-x or click Exit with the stylus.

Links

nano man page

nano-tiny man page

x-debug-tools

Juha Kallioinen — Wed, 09 Apr 2008 09:57:20 +0000

Description

A set of basic X window system debugging tools:

xev: monitor X events to a window
xlsatoms: list interned atoms defined on server
xlsclients: list client applications running on a display
xprop: list properties of the given window
xwininfo: show information abut a given window

Packages

source: x-debug-tools

binary: x-debug-tools

Usage Examples

List all windows on screen:

# xwininfo -tree -root

What application owns the window/dialog/popup covering the screen:

# xprop | grep WM_NAME (and click the window)

Is my window receiving the events it should:

# xwininfo | grep id: (click the window)
# xev -id 0xWINDOW_ID

Links

sp-startup-time

Juha Kallioinen — Wed, 09 Apr 2008 09:56:09 +0000

Description

sp-startup-time provides a library that can be preloaded with LD_PRELOAD to measure the startup time of Gtk applications.

You can use it to identify bottlenecks in your application's startup and possibly to improve the user experience by providing you information to optimize the startup time.

Packages

source: sp-startup-time

binary: sp-startup-time

Usage Examples

How it works

There are a few functions that are typically called during a Gtk application's startup and libstartup intercepts these function calls and collects the time each of them was called. The startup is considered finished once the installed Gtk idle callback is called.

LD_PRELOAD has some shortcomings:

It can be used only if the applications can be run directly, i.e. it does not work with maemo-launched Gtk applications.
Prelink information cannot be used with LD_PRELOAD. This means that prelinked applications spend time in resolving library symbols when this is used.

Note!

In scratchbox LD_PRELOAD does not work directly because scratchbox uses that by itself, use the run-with-startup-time script instead.

If instructed, libstartup will also write a full log of the intercepted calls. This is done at the time the application is unloaded.

Preloading

You can use two methods for preloading the libstartup binary during application execution; environment or global ld.so configuration.

Setting the LD_PRELOAD environment variable like below is preferred as that way it doesn't affect any other programs:

# LD_PRELOAD=/usr/lib/sp-startup-time/libstartup.so app-to-run

However if the "app-to-run" happens to be a launcher script, not a real binary you might need to edit the script itself to contain the preload setting like this:

#!/bin/sh
export LD_PRELOAD=/usr/lib/sp-startup-time/libstartup.so
app-to-run

Apart from the environment you can use the global ld.so configuration to force preloading libstartup for all applications by editing the /etc/ld.so.preload file. This is not however encouraged as any mistake might make the system unbootable.

Configuration

Normally libstartup writes the time application used to reach Gtk idle into syslog, but you can get more detailed information and direct it also to elsewhere with something like this:

export STARTUP_IDLE_QUIT=yes
export STARTUP_WRITE_LOG=yes
export STARTUP_LOG_FILE=/home/user/startup-log.csv

See also "How to interpret the results" below.

The environment variables and how they control the runtime behavior of libstartup are:

STARTUP_LOG_DIR

Explicitly set the directory where libstartup will write the logs.

If not set the logs will be written to the current working directory of the process at the time when libstartup init is called - that is just before the main function is entered.

STARTUP_LOG_FILE

Explicitly set path used for writing logs.

If not set <$STARTUP_LOG_DIR>/<appname>--<pid>.startup will be used.

STARTUP_WRITE_LOG

If set to 'yes', and the process reaches Gtk idle state, the full startup log will be written.

If set to 'force' the startup log will be written always, not just for Gtk apps.

STARTUP_IDLE_QUIT

If set (to anything) gtk_main_quit() will be called when idle state is reached. This is useful for batch testing several runs / applications.

The values can also be set by writing them to /tmp/startup.env. This allows changing configuration for applications for which the environment values would be hard to modify. You could for example use /etc/ld.so.preload to globally enable libstartup use for all applications and then set the configuration options via /tmp/startup.env.

Custom timestamps

Sometimes you might want to add application specific measurement points to the libstartup log. This can be accomplished by inserting code similar to the example below to the application source file(s):

+------- hello.c -------------------------------------------
#include <stdio.h>

void __cyg_profile_func_enter(const void *, const void *);

#define libstartup_custom_timestamp(text) \
  __cyg_profile_func_enter((void *)(-1),text)

int main(void)
{
  sleep(1);

  libstartup_custom_timestamp("main1");
  printf("hello, "); fflush(stdout);
  sleep(1);

  libstartup_custom_timestamp("main2");
  printf("world\n"); fflush(stdout);
  sleep(1);

  libstartup_custom_timestamp("main3");

  sleep(1);
  return 0;
}
+------- hello.c -------------------------------------------

It compiles as usual:

# gcc -o hello hello.c

It executes as normal without libstartup:

# ./hello 
hello, world

And you get the custom timestamps with:

# on device:
$ STARTUP_WRITE_LOG=f LD_PRELOAD=/usr/lib/sp-startup-time/libstartup.so ./hello

# in scratchbox:
$ STARTUP_WRITE_LOG=f run-with-startup ./hello

hello, world

# a file will be created:
+------- hello--5092.startup -------------------------------
generator=sp-startup-time 0.0.7
create_tod=1195657960.406658

t_tod,t_real,t_user,t_sys,f_real,f_user,f_sys,name
1195657960.406658,0.000,0.000,0.000,0.000,0.000,0.000,create
1195657960.406660,0.000,0.000,0.000,0.000,0.000,0.000,lib_init
1195657961.412616,1.000,0.000,0.000,0.000,0.000,0.000,main1
1195657962.422009,2.010,0.000,0.000,0.000,0.000,0.000,main2
1195657963.427050,3.020,0.000,0.000,0.000,0.000,0.000,main3
1195657964.432335,4.020,0.000,0.000,0.000,0.000,0.000,lib_exit
1195657964.432337,4.020,0.000,0.000,0.000,0.000,0.000,lib_fini
+------- hello--5092.startup -------------------------------

The rationale behind hijacking __cyg_profile_func_enter():

we can't override functions defined in the main application binary using LD_PRELOAD -> it must be in some library
we could provide dummy library with some kind of hook, but that would require modifying the application linking, not just the source code

=> we use a dummy function provided by gnu libc

Note: The amount of measurement points is currently limited to 32. For Gtk applications this should leave 20 or so custom entries.

Note: If the same custom tag is used several times, only the last occurence is placed to the log file.

How to interpret the results

syslog

Jan 27 14:33:51 localhost libstartup[12706]: firefox-bin=1.19 seconds
Jan 27 14:34:32 localhost libstartup[12727]: firefox-bin=1.13 seconds

The elapsed wall-clock time from process creation to Gtk idle time is written as seconds.

startup log

Entry time stamp:

t_tod time since epoch.
t_real elapsed time since process create
t_user CPU time spent in user land before call
t_sys CPU time spent in kernel space before call

Time within:

f_real elapsed time within entry
t_user CPU time spent in user land during the call
t_sys CPU time spent in kernel space during the call

Entry identification:

name name of called function, or function1.leave->function2.entry transition

+------- firefox-bin--12727.startup ------------------------
generator=sp-startup-time 0.0.7
create_tod=1195658611.448291

t_tod,t_real,t_user,t_sys,f_real,f_user,f_sys,name
1195658611.448291,0.000,0.000,0.000,0.280,0.000,0.010,create
1195658611.728293,0.280,0.000,0.010,0.000,0.000,0.000,lib_init
1195658611.729729,0.280,0.000,0.010,0.000,0.000,0.000,gtk_init
1195658611.735175,0.280,0.000,0.010,3.010,0.110,0.040,gtk_init->gtk_main
1195658614.744454,3.290,0.110,0.050,0.000,0.000,0.000,gtk_main
1195658614.744454,3.290,0.110,0.050,2.290,0.290,0.030,gtk_main->gui_idle
1195658617.032341,5.580,0.400,0.080,0.000,0.000,0.000,gui_idle
1195658617.032341,5.580,0.400,0.080,18.740,0.120,0.020,gui_idle->gtk_main_quit
1195658635.775691,24.320,0.520,0.100,0.010,0.000,0.000,gtk_main_quit
1195658635.775702,24.330,0.520,0.100,0.080,0.050,0.000,gtk_main_quit->lib_exit
1195658635.855718,24.410,0.570,0.100,0.000,0.000,0.000,lib_exit
1195658635.855720,24.410,0.570,0.100,0.000,0.000,0.000,lib_fini
+------- firefox-bin--12727.startup ------------------------

The following calls are intercepted and logged. The entries with preceding '+' will have time stamps for both entry to and leave from function. The ones marked with '-' will have only entry time stamps.

+ create + lib_init (installs lib_exit as atexit handler) + gtk_init (installs gui_idle as gtk idle handler) - gtk_main - gui_idle + gtk_main_quit - lib_exit - lib_fini

The timestamps are obtained with times() syscall which allows us to trace:

real time (elapsed wall clock time)
user time (time spent executing code)
sys time (time spent in kernel servicing the process)

If there time difference between leaving the previous traced function and entering the next, a transition entry will be written. The interpretation of these transition entries depends on the situation, but generally:

lib_init -> gtk_init

Usually libstartup initialization is executed just before control is yielded to main() entry of the application. So this transition can be considered as "non-gui-init".

gtk_init -> gtk_main

Setting up application main GUI.

gtk_main -> gui_idle

Realizing application main GUI.

gui_idle -> gtk_main_quit

Normal application operation.

gtk_main_quit -> lib_exit

Application shutdown

lib_exit -> lib_fini

As the lib_exit is most likely the last atexit handler called and lib_fini the first library unload code executed, this is very unlikely to have nonzero time.

The "create" entry needs a bit of explanation too. The idea is that this contains time taken by the dynamic linker to bind the various binaries to the process. As the init code from libstartup is if not the last at least one of the last things done before entering application main() function, we can guess that the CPU time spent before this is from the dynamic loader. Perhaps surprisingly determining the real time is quite difficult. The reason is that the syscalls that can be used to query time values return corrected real time and the process create time (as available via /proc/pid/status) contains uncorrected jiffies value. To overcome this the libstartup forks a child process and uses the create time difference between parent and child as elapsed real time estimate. The child process is terminated right after the time difference evaluation.

The idle detection is also not without problems. It is common that there is a brief idle period right after entering gtk_main. For this reason the idle status is is noted only when the idle handler is called enough times without using CPU for other purposes (in the context of the application process).

Links

run-with-startup-time man page

sp-rich-core

Juha Kallioinen — Wed, 09 Apr 2008 09:55:36 +0000

Description

sp-rich-core is a package that installs an init script that pipes core dumps through a script. This script collects lots of information from the system and then saves that and the core dump data compressed into a file (by default onto an MMC card). The collected information includes process SMAPS data, most of /proc data, last lines from syslog, df and ifconfig output etc.

Compressing the core dumps (with lzo) actually speeds up saving them.

Use rich-core-extract (from sp-rich-core-postproc package) to extract the information from a rich core dump.

Packages

source: sp-rich-core

binary: sp-rich-core, sp-rich-core-postproc

Usage Examples

In order to get core dumps, one of these directories needs to exist: /media/mmc1/core-dumps or /media/mmc2/core-dumps.

Create a small program that's sure to crash:

- - - snip crasher.c - - -
int main()
{
   char *p = 0;
   memcpy(p, 0x00, sizeof(p));

}
- - - snap crasher.c - - -

# gcc -g -o crasher crasher.c

After copying the crasher program to your device and having installed the sp-rich-core package, you can now run the program.

Nokia-N810:~# ./crasher
Segmentation fault (core dumped)

Nokia-N810:~# cd /media/mmc1/core-dumps
Nokia-N810:~# rich-core-extract crasher-11-1452.rcore

# by default this will create a directory of the same name without the
# suffix

Nokia-N810:~# ls -l crasher-11-1452/
-rw-r--r-- 1 user root      9 2007-11-19 08:40 cmdline
-rw-r--r-- 1 user root     54 2007-11-19 08:40 component_version
-rw-r--r-- 1 user root 135168 2007-11-19 08:40 coredump
-rw-r--r-- 1 user root     32 2007-11-19 08:40 date
-rw-r--r-- 1 user root    589 2007-11-19 08:40 df
-rw-r--r-- 1 user root    217 2007-11-19 08:40 fd
-rw-r--r-- 1 user root    766 2007-11-19 08:40 ifconfig
-rw-r--r-- 1 user root   1284 2007-11-19 08:40 ls_proc
-rw-r--r-- 1 user root    410 2007-11-19 08:40 osso-product-info
-rw-r--r-- 1 user root     40 2007-11-19 08:40 osso_software_version
-rw-r--r-- 1 user root  28911 2007-11-19 08:40 proc2csv
-rw-r--r-- 1 user root  11983 2007-11-19 08:40 slabinfo
-rw-r--r-- 1 user root   2653 2007-11-19 08:40 smaps

The directory will contain lots of potentially useful information in addition to the core dump file.

Links

rich-core-dumper man page

rich-core-extract man page

sp-error-visualizer

Juha Kallioinen — Wed, 09 Apr 2008 09:55:11 +0000

Description

sp-error-visualizer is a small utility that's designed to help you in noticing errors that are written to syslog.

Packages

source: sp-error-visualizer

binary: sp-error-visualizer

Usage Examples

Just install the sp-error-visualizer package and it will immediately start tracking syslogged messages. It's also started automatically on bootup. If syslog is not running, it provides itself the syslog socket for the applications.

The error visualizer can be disabled by running the following command or uninstalling the package.

# /etc/init.d/sp-error-visualizer stop

The search strings for the errors that are shown from syslog are configured in: /usr/share/sp-error-visualizer/data/syslog-common-errors.conf.

Links

sp-error-visualizer man page

ltrace

Juha Kallioinen — Wed, 09 Apr 2008 09:54:12 +0000

Description

Ltrace traces both system and library calls.

Ltrace is quite similar to strace the main differences being:

ltrace can trace also function calls between shared libraries.
ltrace does not work over clone(). If threading or forking is used, ltrace needs to be attached to the process after that has happened.

Packages

source: ltrace

binary: ltrace

Usage Examples

Attach to the process with process id pid and begin tracing:

# ltrace -p pid

Links

ltrace man page

maemo-debug-scripts

Juha Kallioinen — Wed, 09 Apr 2008 09:53:39 +0000

Description

maemo-debug-scripts is a convenience package for maemo developers needing to debug their programs. The package will:

Post-install a symlink required by Gdb/Valgrind/Oprofile to find the debug symbol files in Scratchbox. The files are installed in /usr/lib/debug/<library path>, but in Scratchbox the library realpaths are /targets/<target name>/<library path>.
Depend from libc6-dbg. Without libc6-dbg threads cannot be debugged with Gdb and Valgrind reports redundant errors for the dynamic loader (its error suppressions for /lib/ld functions can match them only with the debug symbol file as stripped binary doesn't contain static function names)
Depend from target native Gdb. The Scratchbox host version of Gdb (cross-gdb) doesn't know how to load the correct thread library (even on x86) and it cannot debug ARM core-dumps. Native one can.
Install a few helper scripts:
- native-gdb
  
  runs the target native Gdb (by default Scratchbox runs the host binary if it exists, this script overrides that)
- debug-dep-install
  
  installs debug symbols for given binaries and their dependencies listed by "ldd"
- debug-pkg-check
  
  checks the correctness of a debug package for a given package
- list-mmapped-libs
  
  lists all packages containing libs and binaries that given process uses. It can be used to find out the libraries mmap()ed through dlopen that "ldd" (used by debug-deb-install script) doesn't find
- run-with-valgrind
  
  helper script for running a binary with Valgrind memcheck plugin (memory access errors, leaks etc) so that all suitable options are set for it
- run-with-callgrind
  
  helper script for running a binary with Valgrind callgrind plugin (performance profiling) so that all suitable options are set for it
- run-with-massif
  
  helper script for running a binary with Valgrind massif plugin (memory usage visualization) so that all suitable options are set for it
- run-with-strace
  
  helper script for running a binary with strace. Like above helper scripts, handles maemo-launched binaries automatically

Packages

source: maemo-debug-scripts

binary: maemo-debug-scripts

Usage Examples

Links

debug-dep-install man page

debug-pkg-check man page

list-mmapped-libs man page

native-gdb man page

run-with-callgrind man page

run-with-massif man page

run-with-memcheck man page

run-with-strace man page

wireless-tools

Juha Kallioinen — Wed, 09 Apr 2008 09:48:46 +0000

Description

This package contains the Wireless tools, used to manipulate the Linux Wireless Extensions. The Wireless Extension is an interface allowing you to set Wireless LAN specific parameters and get the specific stats.

ifrename - rename network interfaces based on various criteria.
iwconfig - configure a wireless network interface.
iwevent - display wireless events generated by drivers and setting changes.
iwgetid - report ESSID, NWID or AP/Cell Address of wireless network.
iwlisst - get more detailed wireless information from a wireless interface.
iwpriv - configure optional parameters of a wireless network interface.
iwspy - Get wireless statistics from specific nodes. NOTE: This tool does not currently work!

Packages

source: wireless-tools

binary: wireless-tools, ifrename

Usage Examples

Links

bluez-hcidump

Juha Kallioinen — Wed, 09 Apr 2008 09:48:19 +0000

Description

Bluez-hcidump is the BlueZ project Bluetooth HCI data dumper.

Hcidump lets you view raw HCI data to and from a Bluetooth device in a human readable way.

Packages

source: bluez-hcidump

binary: bluez-hcidump

Usage Examples

Start hcidump like this and create a Bluetooth connection to your phone:

# hcidump -V -x

you'll see a lot of verbose messages of the data and state of the connection.

Links

hcidump man page

http://www.bluez.org/

xnee

Juha Kallioinen — Wed, 09 Apr 2008 09:47:37 +0000

Description

Xnee is a suite of programs that can record, replay and distribute user actions (X events) under the X11 environment. It can be used for example for test automation.

Xnee installs two programs; cnee and gnee. cnee is the command line utility and gnee is the graphical user interface to cnee.

Limitation: Xnee does not support the XInput extension used for thumb interaction.

Packages

source: xnee

binary: xnee

Usage Examples

Record mouse events:

# cnee --record --mouse --events-to-record 100 -o mouse_rec.xnl

Do some actions with the stylus and press control-C once you're done, or continue using the stylus until 100 events are reached at which point cnee will exit.

To replay back what you just recorded:

# cnee --replay --file mouse_rec.xnl&

Links

cnee man page

http://www.gnu.org/software/xnee/

http://www.linuxjournal.com/article/6660

sp-stress

Juha Kallioinen — Wed, 09 Apr 2008 09:47:05 +0000

Description

sp-stress contains utilities that can create artificial system load. This can be useful when you want to verify that your application works also when the system is being otherwise stressed.

The utilities are:

cpuload for generating CPU load
ioload for generating I/O load
memload for generating memory load

Packages

source: sp-stress

binary: sp-stress

Usage Examples

To create a 30% CPU load on the system:

# cpuload 30

The I/O load is generated by performing read and write operations on a file that has been created just for this purpose. The file has to be on one of the MMC's since the device's internal flash memory is unsuitable for these operations (see the man page for more info).

Note!

The contents of the file will be *destroyed*, so do not do this on a file that you care about.

# ioload workfile

To consume 20 megabytes of memory:

# memload 20

Links

cpuload man page

ioload man page

memload man page

xrestop

Juha Kallioinen — Wed, 09 Apr 2008 09:46:07 +0000

Description

Xrestop shows how much X resources from the X server are being used by the X clients. It is similar to top and htop.

Packages

source: xrestop

binary: xrestop

Usage

# xrestop

Links

xrestop man page

http://www.freedesktop.org/wiki/Software/xrestop

sysstat

Juha Kallioinen — Wed, 09 Apr 2008 09:44:37 +0000

Description

The sysstat package contains the sar, mpstat, iostat, sadf and sa tools for Linux.

sar command collects and reports system activity information.
iostat command reports CPU utilization and I/O statistics for disks.
mpstat command reports global and per-processor statistics.
sadf command is used to display data collected by sar in various formats.

The statistics reported by sar concern I/O transfer rates, paging activity, process-related activities, interrupts, network activity, memory and swap space utilization, CPU utilization, kernel activities and TTY statistics, among others.

Packages

source: sysstat

binary: sysstat

Usage Examples

Links

http://perso.orange.fr/sebastien.godard/

chinook

chinook

sp-endurance

Description

Packages

Usage

Emphasizing leakage

Post-processing

Links

See Also

oprofile

Description

Packages

Installing Oprofile

Configuring the device

Getting a new kernel

Using the new kernel

Restoring the normal kernel

Recognizing a suitable kernel

Installing oprofile to the device

Installing debug symbols

Usage

Viewing profile reports

Profiling with callgraphs

Viewing reports from a PC

Oprofile with kcachegrind

Links

See Also

traceroute

Description

Packages

Usage Examples

Links

See Also

netcat

Description

Packages

Usage Examples

Links

See Also

iputils

Description

Packages

Usage Examples

Links

See Also

xresponse

Description

Packages

Usage Examples

Monitoring screen updates

Timing application response / screen updates

Links

See Also

nano

Description

Packages

Usage Examples

Links

See Also

x-debug-tools

Description

Packages

Usage Examples

Links

See Also

sp-startup-time

Description

Packages

Usage Examples

How it works

Preloading

Configuration

Custom timestamps

How to interpret the results

syslog

startup log

Links

See Also

sp-rich-core