Maemo Debugging Guide

Introduction

This debugging guide is targeted for beginner level maemo developers who need to know how to do debugging in the maemo environment.

In this document we cover the three basic debugging tools that are available in the maemo environment and show how to use them. The tools are:

• gdb - The Gnu Project Debugger. General tool for various debugging needs.
• strace - System call tracer. This tool prints out all the system calls made by the traced process.
• valgrind - Debugger and profiler. Valgrind works only in the X86 environment under Scratchbox so this tool can not be used in the device itself.

This document assumes that the developer (you!) already knows how to:

• do software development in the Linux environment using the C-language
• install software to the Tablet device
• how to gain root access to the device
• how to install osso-xterm and ssh to the device
• configure repositories in the /etc/apt/sources.list file
• setup USB networking between your Linux PC and the Tablet (you can -of course- just use the device over your local WLAN connection instead)
• work with the Scratchbox environment and Scratchbox targets

We do not explain in this document in detail how to do any of the above mentioned things but instead we might just say that you need to install ssh or you need to install osso-xterm to the device and it is assumed that you know how to do this. If you do not know how to do these things then please first read the other documents available in the maemo.org site and in the maemo wiki.

Prerequisites

To follow the debugging examples in this document you need to have:

• maemo 3.x 'bora' SDK with Scratchbox 1.X installed in your Linux PC
• Nokia Internet Tablet device running one of the IT OS 2007.X releases
• USB cable to connect the device with the Linux PC
• Internet access both for the Tablet and for your Linux PC
• USB networking (or WLAN) setup between the Linux PC and the device
• Root login access to the device over ssh
• osso-xterm installed in the device
• ssh software installed in the device

General notes about debugging

Don't forget to install the debug symbols for the widely used C-library itself. If you do debugging in the Internet Tablet device you need to install the libc6-dbg package in the device. You can do this by running apt-get install libc6-dbg in the Internet Tablet device itself.

Debugging issues on the ARM architecture

There are some issues you need to know and be aware when you are debugging on the ARM architecture.

• To make backtraces work properly in ARM side you need to install the dbg packages of the libraries your application is using. Profiling and debugging (gdb) tools require code to have either framepointers or debugging symbols to unwind stack. This is needed for showing backtraces or call graphs.
• C-language functions with the __attribute__((__noreturn__)) statement need to be compiled with the gcc option: -fno-omit-frame-pointer. Without framepointers you can not get backtrace through "noreturn" functions. In practice, what would happen is that when you use the bt command you would see infinite repeat of this kind of function.
• In addition, for the gdb to be able to display correct function names during debugging it also needs to have access to the debug symbols. Without them it shows for the given address the preceding exported function name.

Debugging issues in Scratchbox

For debuggers and profilers (gdb, valgrind, oprofile ...) to find the debug symbol files from directory under /usr/lib/debug corresponding to the directory of the original file you need to have symlinks for this in scratchbox.

For example, the libc file is:

[sbox-SDK_X86: ~] > ls -l /lib/libc-2.3.6.so 
-rwxr-xr-x  1 user user 1164796 Nov 20 14:33 /lib/libc-2.3.6.so
	

And it's debug symbols file (after doing apt-get install libc6-dbg) is installed in:

[sbox-SDK_X86: ~] > ls -l /usr/lib/debug/lib/libc-2.3.6.so 
-rwxr-xr-x  1 user user 318068 Nov 20 14:33 /usr/lib/debug/lib/libc-2.3.6.so
	

However, in Scratchbox the real path to libc is:

[sbox-SDK_X86: ~] > realpath  /lib/libc-2.3.6.so 
/targets/SDK_X86/lib/libc-2.3.6.so
	

Therefore some debuggers might search the debug symbols from /usr/lib/debug/targets/SDK_X86/lib/libc-2.3.6.so and clearly this would fail. To solve this you need to set a symlink like this:

[sbox-SDK_X86: ~] > mkdir -p /usr/lib/debug/targets
[sbox-SDK_X86: ~] > cd /usr/lib/debug/targets
[sbox-SDK_X86: /usr/lib/debug/targets ] > ln -sf /usr/lib/debug \
               $(sh -c ". /targets/links/scratchbox.config;"'echo $SBOX_TARGET_NAME')
	

After this you should have the path set correctly for libc so that debuggers can find its debug symbols.

In general it is recommended to use the native gdb in the target and not the Scratchbox host-gdb.

If you need to debug threads in your application you need to use gdb that is linked against the same thread library that your application is using. For this reason the Scratchbox provided gdb is not suitable for threads debugging so use the native gdb. See instructions in the next chapter how to start using the native gdb.

Using the gdb debugger

Introduction to gdb

The Gnu Project Debugger, or gdb for short, is a general purpose debugger that can be used for various debugging purposes.

This section does not explain how to use the gdb debugger itself ie we do not explains what specific commands in gdb perform some specific actions. There are other tutorials and documentation readily available in the Internet (http://www.gnu.org/software/gdb/documentation/) for that purpose and for this reason this document focuses to explain how to setup and perform the basic debugging steps with the gdb in the maemo environment.

Setting up the environment

You need to setup both the Internet Tablet device and the Scratchbox environment.

Preparing the Scratchbox environment for debugging

If you have not yet installed maemo 3.x bora release and Scratchbox Apophis to your Linux PC then install these first before continuing. Refer to the maemo 3.x INSTALL.txt file (available in the downloads section for maemo 3.x in maemo.org) for details how to do this.

After installing maemo 3.x 'bora' the default target names for Armel and X86 are SDK_ARMEL and SDK_X86. We will use these target names in this example.

Next, install gdb to the Scratchbox from the maemo 3.x 'bora' repositories.

[sbox-SDK_X86: ~] > apt-get install gdb
Reading Package Lists... Done
Building Dependency Tree... Done
The following NEW packages will be installed:
gdb
0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
Need to get 1176kB of archives.
After unpacking 2662kB of additional disk space will be used.
Get:1 http://repository.maemo.org bora/free gdb 6.4-osso2 [1176kB]
Fetched 1176kB in 4s (270kB/s)
Selecting previously deselected package gdb.
(Reading database ... 14922 files and directories currently installed.)
Unpacking gdb (from .../gdb_6.4-osso2_i386.deb) ...
Setting up gdb (6.4-osso2) ...
[sbox-SDK_X86: ~] > 
	  

Now you have two gdb programs installed in your environment. To see which one is used just check it with:

[sbox-SDK_X86: ~] > which gdb
/targets/links/arch_tools/bin/gdb
	  

As you can see, if you just say gdb then the gdb used is the one provided by Scratchbox.

Start briefly both gdb debuggers and see that they start properly in the scratchbox environment. First just run gdb:

[sbox-SDK_X86: ~] > gdb
GNU gdb 6.4.90
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu".
(gdb) q
[sbox-SDK_X86: ~] > 
	  

Then run the maemo version of gdb:

[sbox-SDK_X86: ~] > SBOX_REDIRECT_IGNORE=/usr/bin/gdb /usr/bin/gdb
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux".
(gdb) q
[sbox-SDK_X86: ~] > 
	  

You can quit from gdb by typing 'q' and hit ENTER.

We will always use the native gdb in this document for the reason that with that you can also debug threads.

For this purpose, let's create an alias for our native gdb. Do this:

[sbox-SDK_X86: ~] > alias gdb='SBOX_REDIRECT_IGNORE=/usr/bin/gdb run-standalone.sh /usr/bin/gdb'
	

Now when you just give the gdb command it should start the native gdb.

You should now have both gdb versions installed in your Scratchbox X86 environment. After setting the alias as shown above you should always get the native gdb instead of the Scratchbox version.

Preparing the Internet Tablet for debugging

You need to install gdb into your Internet Tablet device. It is recommended that you first install osso-xterm and then ssh into your device before continuing.

• 1. Install osso-xterm to your device and start it.
• 2. Gain root access to the device while running osso-xterm. See: https://maemo.org/maemowiki/HowDoiBecomeRoot2 how to gain root access.
• 3. Edit the /etc/apt/sources.list file in the device so that it contains a line:

  deb http://repository.maemo.org/ bora free non-free
  	      
  

  You can edit the /etc/apt/sources.list file with a text editor (like vi) but you could also simply give the below command from the osso-xterm:

  echo "deb http://repository.maemo.org/ bora free non-free" >> /etc/apt/sources.list
  	      
  
  

  Important: it is not recommended to do device software updates from the maemo sdk repositories (for example, do not do 'apt-get upgrade' in the device). The reason for this is that there might be some software packages in the SDK repositories that are so called sdk variants and they might create a problem if directly installed in the actual device. In this example we only download gdb software from the repository.
• 4. Next, do a apt-get update in your device (not apt-get upgrade !!). The update command will refresh the packagelist database in the device.
• 5. Next, do a apt-get install gdb
• 6. Next, install the ssh software to the device, do a apt-get install ssh in the device.

You should now have the gdb, gdbserver (included in the gdb package) and ssh programs installed in your device.

Debugging use cases with gdb

Debugging a command line application in Scratchbox X86 environment with gdb

One of the most common debugging case is doing debugging in the Scratchbox X86 environment.

By now, you should already have the gdb installed in your SDK_X86 target. Next, download the gdb example package from stage.maemo.org. See link in the end of this document. Extract the gdb_example.tar.gz to some directory under your Scratchbox. For example, do the following:

[sbox-SDK_X86: ~] > mkdir src
[sbox-SDK_X86: ~] > cd src/testing

# download gdb_example.tar.gz from stage.maemo.org. See link in the end of the document.
# Copy gdb_example.tar.gz to this directory.

[sbox-SDK_X86: ~/src] > tar xvzf gdb_example.tar.gz
gdb_example/gdb_example.c
gdb_example/gdb_example2.c
[sbox-SDK_X86: ~/src] > cd gdb_example
[sbox-SDK_X86: ~/src/testing/gdb_example] > 
	    

You should have two files: gdb_example.c and gdb_example2.c files now under your ~/src/testing/gdb_example/ directory.

The example apps are:

• the gdb_example.c is a very simple C application that just has some functions that call each other in a row. We use this to demonstrate how to get backtraces.
• the gdb_example2.c is a simple variant of the gdb_example.c that has some additional sleep() calls and we will use this to demonstrate simple core dump debugging.

Next, compile the small gdb_example.c file as shown below and start the gdb debugger. This simple example shows how to set breakpoints and how to get a backtrace from the program. Backtrace tells you what functions have been called and with what parameters.

Compile the gdb_example.c application with the -g option like this:

[sbox-SDK_X86: ~/src/testing/gdb_example] > gcc gdb_example.c -o gdb_example -g 

Next, start the gdb (native, if you set the alias as explained above) with the gdb_example application as a parameter.

[sbox-SDK_X86: ~/src/testing/gdb_example] > gdb gdb_example
GNU gdb 6.4
Copyright (C) 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-linux"...Using host libthread_db library "/lib/libthread_db.so.1".

Set the breakpoint (br) to function example_3

(gdb) br example_3
Breakpoint 1 at 0x80483e9: file gdb_example.c, line 40.

Run the application under gdb so that you give a command line parameter jassoo to the application.

(gdb) run jassoo
Starting program: /home/user/src/testing/gdb_example/gdb_example jassoo
Now in main().
Now in example_1(2) function.
Now in example_2(parameter here) function.

Breakpoint 1, example_3 () at gdb_example.c:40
40        printf("Now in example_3()\n");

From above you can see that the running of the application stopped in function example_3. This happened because you set the breakpoint (br example_3) above.

Now you can list the backtrace (bt) from your application and see what functions have been called. The list goes from recent to older. The oldest function was naturally the main() function in the end of the list. You can also see what were the parameters to the called functions.

(gdb) bt
#0  example_3 () at gdb_example.c:40
#1  0x080483dc in example_2 (a=0x8048528 "parameter here") at gdb_example.c:31
#2  0x080483b9 in example_1 (x=2) at gdb_example.c:22
#3  0x08048387 in main (argc=2, argv=0xbf985944) at gdb_example.c:10
(gdb) 

It is convenient to see what the source code is for some line mentioned in the output. You can for example do:

(gdb) list 31
26       *
27       */
28      int example_2(char* a)
29      {
30        printf("Now in example_2(%s) function.\n",a);
31        example_3();
32        return 0;
33      }
34      /*
35       *
(gdb) 

You can also inspect the value of the variables. For example, do:

(gdb) br example_2
Breakpoint 1 at 0x80483c4: file gdb_example.c, line 30.
(gdb) run
Starting program: /home/user/src/testing/gdb_example 
Now in main().
Now in example_1(1) function.

Breakpoint 1, example_2 (a=0x8048528 "parameter here") at gdb_example.c:30
30        printf("Now in example_2(%s) function.\n",a);

To see the value of variable 'a' just type:

(gdb) print a
$1 = 0x8048528 "parameter here"

(gdb) 

Essentially debugging with gdb in the Scratchbox X86 target is similar than debugging with gdb in any Linux host. In this example we have only used a small subset of gdb's functionality.

Debugging command line application in the Internet Tablet device

It is possible to debug your application in the internet tablet device itself using gdb. Before starting first login to the device and install (if you have not done so yet) the gdb debugger to the device:

Nokia-N800-01 ~# apt-get install gdb
... etc ...
Nokia-N800-01 ~# 
	  

Here we assume that you already have ssh, and osso-xterm installed in the your Tablet and that you can login to the device using ssh from your Linux PC. In addition, you have set the maemo 3.x Bora repository entries in the /etc/apt/sources.list file as explained previously.

Debugging with the gdb debugger in the device is similar than using gdb in a normal Linux PC environment. The limitations are mostly related to the available free RAM memory which in the worst case means that you might run out of memory while trying to debug your application in the device.

In this example we follow the basic logic of the first example but this time we do it in the device.

• First, compile the gdb_example.c application in the scratchbox for armel architecture.

  [sbox-SDK_X86: ~] > sb-conf select SDK_ARMEL
  Hangup
  Shell restarting...
  [sbox-SDK_ARMEL: ~] > pwd
  /home/user
  [sbox-SDK_ARMEL: ~] > cd src/testing/gdb_example
  [sbox-SDK_ARMEL: ~/src/testing/gdb_example] > gcc gdb_example.c -o gdb_example -g
  [sbox-SDK_ARMEL: ~/src/testing/gdb_example] > file gdb_example
  gdb_example: ELF 32-bit LSB executable, ARM, version 1 (SYSV), for GNU/Linux 2.4.17, \
               dynamically linked (uses shared libs), not stripped
  [sbox-SDK_ARMEL: ~/src/testing/gdb_example] > 
  	      
  
  

• Next, copy the armel version of the gdb_example to the tablet. You (of course) need to have the sshd daemon up-and-running in the device before you can copy files with scp.

  [sbox-SDK_ARMEL: ~/src/testing/gdb_example] > scp gdb_example someuser@192.168.2.15:
  someuser@192.168.2.15's password:  ...........
  gdb_example                                           100% 8454     8.3KB/s   00:00    
  
  

• Next, login to your device with ssh. Here we use the name someuser (or just user) as an example only. Fill in your own username that you use in the device (if you have set that). The IP address is an example and your device IP address could be different.

  [sbox-SDK_ARMEL: ~/src/testing/gdb_example] > ssh someuser@192.168.2.15
  someuser@192.168.2.15's password: ..........
  
  BusyBox v1.1.3 (Debian 3:1.1.3-3.sdk2) Built-in shell (ash)
  Enter 'help' for a list of built-in commands.
  
  Nokia-N800 $ pwd
  /home/someuser
  Nokia-N800 $ mkdir src 
  Nokia-N800 $ mkdir src/testing
  Nokia-N800 $ mv gdb_example src/testing
  Nokia-N800 $ cd src/testing
  Nokia-N800 $ pwd
  /home/someuser/src/testing
  
  

• Next, start the gdb debugger with the gdb_example application.

  Nokia-N800 $ gdb gdb_example
  GNU gdb 6.4
  Copyright 2005 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "arm-linux-gnueabi"...Using host libthread_db library "/lib/libthread_db.so.1".
  
  (gdb) 
  
  

  You should now be able to debug this small example application with the similar way than you debugged it in the example above.

Debugging core files in the device

This chapter explains how to debug core files in the Internet Tablet.

The kernel does not dump cores of a failing process unless you have told it where to dump the core files.

You can see what the default values for core dumps are by reading the file linuxrc with the more command in the device and by studying the enable_coredumps() function:

Nokia-N800-01:~# more /mnt/initfs/linuxrc 
... snip ...
enable_coredumps()
{
        coredir=/media/mmc1/core-dumps
        echo -n "Enabling core dumps to $coredir/..."
        echo "$coredir/%e-%s-%p.core" > /proc/sys/kernel/core_pattern
        ulimit -c unlimited
        echo "done."

}
... snip ...
	      

As you can see the default location for core dumps is in /media/mmc1/core-dumps directory. The second echo command defines the name of the core dump file. The default name contains the name of the executable (%e), the signal (%s) and the PID number (%p).

If you want to use these default settings then just create the core-dumps directory under the /media/mmc1 directory.

However, in this example we have a MMC card mounted in the /mnt/somedir directory and we will instruct the kernel to dump core files under this directory with a slightly different core_pattern (see below).

Run these commands in the device as a root:

Nokia-N800-01:~# mkdir /mnt/somedir/core-dumps
Nokia-N800-01:~# echo "/mnt/somedir/core-dumps/core.%e.%p" > /proc/sys/kernel/core_pattern 
	    

Next, we will use our small example application gdb_example2 to demonstrate how to debug the core file.

• Compile the gdb_example2 in the scratchbox SDK_ARMEL target and just copy the file to the device using scp. Then start the gdb_example2 like this:

  Nokia-N800-01:/home/user# ./gdb_example2 &
  Nokia-N800-01:/home/user# gdb_example2.
  Now in main().
  Now in example_1(1) function.
  
  	    
  

• The gdb_example2 is now running in the background and it starts to dump its output to the screen. There are some sleep() calls in the gdb_example2 so that you have time to kill it with the SIGSEGV signal. So, lets just make it generate a core dump. Assuming that the process is referred as %1 just use the kill command as below and hit couple of times the ENTER key:

  
  Nokia-N800-01:/home/user# kill -SIGSEGV %1
  Nokia-N800-01:/home/user# 
  [1] + Segmentation fault (core dumped) ./gdb_example2
  		
  

• You should now have a core file under the /mnt/somedir/core-dumps directory starting with the name core plus including the name of the file and ending with the PID number of the gdb_example2 program. Check that you got it (the 11437 number below will -of course- be different in your environment):

  
  Nokia-N800-01:/home/user# ls -l /mnt/somedir/core-dumps/
  -rw-------    1 root     root       135168 Mar 20 12:26 core.gdb_example2.11437
  Nokia-N800-01:/home/user# 
  		
  

• The gdb_example2 is linked against the libc library and if you want to be able to resolve symbols during debugging also for the library then you need to install libc6-dbg package in the device. This same rule applies to other libraries that your application might be linked against. See the further notes about the DBG packages in this document.

  Let's install the libc6-dbg package to get symbols for the library.

  Nokia-N800-01:/home/user#  apt-get install libc6-dbg
  Reading Package Lists... Done
  Building Dependency Tree... Done
  The following NEW packages will be installed:
    libc6-dbg
  
  .... snip ...
  
  Nokia-N800-01:/home/user# 
  

• Now you can debug the core file together with the gdb_example2 binary that you compiled with the -g flag. Lets try to see where the execution of the gdb_example2 was when you hit it with the -SIGSEGV signal. Start the gdb and give the gdb_example2 as a parameter and the core file as the second parameter:

  
  Nokia-N800-01:/home/user# gdb ./gdb_example2 /mnt/somedir/core-dumps/core.gdb_example2.11437 
  
  GNU gdb 6.4
  Copyright 2005 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "arm-linux-gnueabi"...Using host libthread_db library "/lib/libthread_db.so.1".
  
  Core was generated by `./gdb_example2'.
  Program terminated with signal 11, Segmentation fault.
  Reading symbols from /lib/libc.so.6...BFD: /usr/lib/debug/lib/libc-2.3.6.so: warning: sh_link not set for section `.ARM.exidx'
  Reading symbols from /usr/lib/debug/lib/libc-2.3.6.so...done.
  done.
  Loaded symbols for /lib/libc.so.6
  Reading symbols from /lib/ld-linux.so.3...BFD: /usr/lib/debug/lib/ld-2.3.6.so: warning: sh_link not set for section `.ARM.exidx'
  Reading symbols from /usr/lib/debug/lib/ld-2.3.6.so...done.
  done.
  Loaded symbols for /lib/ld-linux.so.3
  #0  0x400a55c0 in nanosleep () from /lib/libc.so.6
  

  You can see from the example above that now gdb is using debug symbols from /usr/lib/debug/lib/libc-2.3.6.so. If you did not install the libc6-dbg package then gdb would not have information available about the libraries debug symbols.

  Now, the gdb is waiting your command so just give the bt (backtrace) command. You should see something similar to this:

  (gdb) bt
  #0  0x400a55c0 in nanosleep () from /lib/libc.so.6
  #1  0x400a53d4 in sleep () from /lib/libc.so.6
  #2  0x00008424 in example_1 (x=1) at gdb_example2.c:23
  #3  0x000083e0 in main (argc=1, argv=0xbe8506f4) at gdb_example2.c:11
  (gdb) 
  

  For this simple example we installed the available libc6-dbg package before starting to debug the gdb_example2 application. If you do not have the dbg packages installed for various libraries then this would mean that f.ex. backtrace information that comes from the non-debug version of the library can not be trusted.

  Depending at what point you gave the kill -SIGSEGV command you might have different output. In this example we hit the process when it was calling sleep function inside the example_1 function in file gdb_example2.c. We also see that the sleep() function has further called the nanosleep() function and that is when it got the -SIGSEGV signal (see note information above).

Debugging core file from the device inside Scratchbox

Because binaries and libraries in the device are prelinked then doing debugging succesfully inside the Scratchbox environment using the programs core file (from the device) would require that:

• you copy the relevant libraries (that your application is using) from the device to the Scratchbox. Otherwise addresses in the prelinked and non prelinked libraries would not match and therefore gdb backtraces would not load the library.
• If you decide to do core file debugging in the Scratchbox ARMEL environment you need to use the native gdb program instead of the one provided by Scratchbox. See the previous chapter how to set the native gdb as the default one.

After you have copied the /lib/libc6 library from the device and you are using native gdb you can debug the application normally.

Debugging UI applications in Scratchbox X86

Many maemo applications use the graphical UI and debugging these applications differs slightly from debugging simple command line applications.

Here we will use the maemopad as an example application to debug in the X86 target with gdb.

If you have setup your Scratchbox environment correctly as explained above you should be able to follow these steps to do UI debugging with maemopad application.

First, activate the X86 target, go to the testing directory and download the source package of maemopad application.

• [sbox-SDK_ARMEL] sb-conf select SDK_X86
  [sbox-SDK_X86] cd ~/src/
  [sbox-SDK_X86 ~/src] mkdir maemopad 
  [sbox-SDK_X86 ~/src/maemopad] cd maemopad 
  [sbox-SDK_X86 ~/src/maemopad] apt-get source maemopad
  Reading Package Lists... Done
  Building Dependency Tree... Done
  Need to get 26.9kB of source archives.
  Get:1 http://repository.maemo.org bora/free maemopad 1.5 (dsc) [395B]
  Get:2 http://repository.maemo.org bora/free maemopad 1.5 (tar) [26.5kB]
  Fetched 26.9kB in 0s (38.3kB/s) 
  dpkg-source: warning: extracting unsigned source package (./maemopad_1.5.dsc)
  dpkg-source: extracting maemopad in maemopad-1.5
  dpkg-source: unpacking maemopad_1.5.tar.gz
  

• Next, check that you have the right files.

  [sbox-SDK_X86: ~/src/maemopad] > ls -l 
  total 36
  drwxr-xr-x  6 user user  4096 Jun  1  2006 maemopad-1.5
  -rw-rw-r--  1 user user   395 Jan  7 14:51 maemopad_1.5.dsc
  -rw-rw-r--  1 user user 26465 Jan  7 14:51 maemopad_1.5.tar.gz
  
  

• Next, enter to the maemopad-1.5 source directory and set the DEB_BUILD_OPTIONS environment variable so that the generated binaries are not stripped and then build the maemopad package with dpkg-buildpackage command as shown here:

  [sbox-SDK_X86: ~/src/maemopad] > cd maemopad-1.5
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > export DEB_BUILD_OPTIONS=debug,nostrip
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > dpkg-buildpackage -rfakeroot
  dpkg-buildpackage: source package is maemopad
  dpkg-buildpackage: source version is 1.5
  dpkg-buildpackage: source changed by Maemo Integration <integration@maemo.org>
  dpkg-buildpackage: host architecture i386
  dpkg-buildpackage: source version without epoch 1.5
  dpkg-checkbuilddeps: Using Scratchbox tools to satisfy builddeps
  .... etc ....
  
  

  In this example we use the standard export DEB_BUILD_OPTIONS=debug,nostrip environment variable but there might be source packages that do not support these debug,nostrip options. In this case you need to make sure that the source is compiled with -g flag (usually this option can be added to the CFLAGS variable in the debian/rules file) and that the produced binaries will not be stripped.
  
  In the long term it is better to modify the source package to generate a separate debug symbol (-dbg) package. This requires that you modify both the debian/rules and debian/control files.
• You should now have maemopad binaries generated so that they have the debug symbols in them. Check that you get "not stripped" flag from the maemopad binary:

  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > file debian/tmp/usr/bin/maemopad 
  debian/tmp/usr/bin/maemopad: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for \
   GNU/Linux 2.0.0, dynamically linked (uses shared libs), not stripped
  

• You should now have a maemopad_1.5_i386.deb file in the ~/src/maemopad directory. Check that you have it:

  [sbox-SDK_X86: ~/src/maemopad] > ls -l 
  total 80
  drwxr-xr-x  6 user user  4096 Mar 13 09:52 maemopad-1.5
  -rw-rw-r--  1 user user   395 Mar 13 09:52 maemopad_1.5.dsc
  -rw-rw-r--  1 user user 26459 Mar 13 09:52 maemopad_1.5.tar.gz
  -rw-rw-r--  1 user user   727 Mar 13 09:53 maemopad_1.5_i386.changes
  -rw-r--r--  1 user user 38266 Mar 13 09:53 maemopad_1.5_i386.deb
  [sbox-SDK_X86: ~/src/maemopad] > 
  
  

• Next, install the newly compiled maemopad_1.5_i386.deb file inside the scratchbox environment:

  [sbox-SDK_X86: ~/src/maemopad] > dpkg -i maemopad_1.5_i386.deb
  ... output from dpkg ...
  

• Next, start the Xephyr server outside the Scratchbox:

  Linux-PC $ Xephyr :2 -host-cursor -screen 800x480x16 -dpi 96 -ac &
  

• Next, start the Application Framework from inside the scratchbox X86 target:

  
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > export DISPLAY=:2
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > af-sb-init.sh start &
  ... lots of output from various programs ... 
  

  You should now have the application framework up and running and the Xephyr window should contain the normal SDK UI.
• Next, you need to move to the source directory:

  
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5] > cd src/ui
  

• In the ui directory you should have the files:

  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5/src/ui] ls -l 
  total 36
  -rw-r--r--  1 user user  9377 Jun  1  2006 callbacks.c
  -rw-r--r--  1 user user  1857 Jun  1  2006 callbacks.h
  -rw-r--r--  1 user user 14769 Jun  1  2006 interface.c
  -rw-r--r--  1 user user  3157 Jun  1  2006 interface.h
  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5/src/ui] > 
  

• In this example we will set a debugging breakpoint at the callback function callback_help() that is located in the callbacks.c file. This function is called when the user clicks the HELP button from the maemopad menu. You set the debugging breakpoint like this:

  [sbox-SDK_X86: ~/src/maemopad/maemopad-1.5/src/ui] > gdb maemopad
  GNU gdb 6.4
  Copyright (C) 2005 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "i386-linux"....Using host libthread_db library "/lib/libthread_db.so.1".
  
  (gdb) br callback_help
  Breakpoint 1 at 0x804bd86: file ui/callbacks.c, line 206.
  
  

• Next, start the maemopad application from the gdb.

  (gdb) run
  Starting program: /targets/SDK_X86/usr/bin/maemopad 
  /home/user/.osso/current-gtk-key-theme:1: Unable to find include file: "keybindings.rc"
  maemopad[32714]: GLIB WARNING ** GLib-GObject - invalid cast from `HildonProgram' to `GtkWidget'
  maemopad[32714]: GLIB CRITICAL ** Gtk - gtk_widget_queue_resize: assertion `GTK_IS_WIDGET (widget)' failed
  maemopad[32714]: GLIB WARNING ** GLib-GObject - gsignal.c:2139: signal id `7' is invalid for instance `0x8061e20'
  maemopad[32714]: GLIB WARNING ** GLib-GObject - IA__g_object_notify: object class `HildonProgram' has no property named `visible'
  

  Thread-debugging: We are now using here the native gdb program. If you need to debug threads in your application you need to use the native gdb that is linked against the same thread library that your application is using. Keep in mind that Scratchbox gdb does not fit to this purpose.
  
  To use the native gdb you need to use SBOX_REDIRECT_IGNORE as mentioned elsewhere in this document.
• Now you should be able to see the maemopad application inside the Xephyr window and you should be able to use it normally. Next, click the upper menu and select the item HELP. In your gdb terminal window you should now see :

  Breakpoint 1, callback_help (action=0x805e768, data=0x8067170) at ui/callbacks.c:206
  206     {
  (gdb) 
  

• The breakpoint that you set above is now reached and execution of maemopad application is stopped. The gdb debugger waits your command now. Try something simple, like use the list command to see where is the execution of the application going. You should get:

  (gdb) list
  201         }
  202     }
  203
  204     /* help */
  205     void callback_help( GtkAction * action, gpointer data )
  206     {
  207         osso_return_t retval;
  208
  209         /* connect pointer to our MainView struct */
  210         MainView *mainview = NULL;
  (gdb) 
  

• You can now debug the maemopad normally. Lets try to see if we could execute the maemopad step by step so that we could get the help window on screen. Give the s command like this:

  (gdb) s
  212         g_assert(mainview != NULL && mainview->data != NULL );
  (gdb) s
  214         retval = ossohelp_show(
  (gdb) s
  
  0xb778aa38 in g_cclosure_marshal_VOID__VOID () from /usr/lib/libgobject-2.0.so.0
  (gdb)
  

  After the last 's' command the HELP window should pop-up in the maemopad. You need to click the CLOSE button to continue because execution is now in the UI framework.
• You could also try the bt (backtrace) command to see what functions were called. You should see a list of functions that have been called like this:

  (gdb) bt
  #0  0xb778aa38 in g_cclosure_marshal_VOID__VOID () from /usr/lib/libgobject-2.0.so.0
  #1  0xb777429b in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
  #2  0xb7789408 in g_signal_has_handler_pending () from /usr/lib/libgobject-2.0.so.0
  #3  0xb778a358 in g_signal_emit_valist () from /usr/lib/libgobject-2.0.so.0
  #4  0xb778a646 in g_signal_emit () from /usr/lib/libgobject-2.0.so.0
  #5  0xb7dd1ccb in gtk_widget_activate () from /usr/lib/libgtk-x11-2.0.so.0
  #6  0xb7ce2900 in gtk_menu_shell_activate_item () from /usr/lib/libgtk-x11-2.0.so.0
  #7  0xb7ce2dbb in gtk_menu_shell_activate_item () from /usr/lib/libgtk-x11-2.0.so.0
  #8  0xb7cd878e in gtk_menu_reorder_child () from /usr/lib/libgtk-x11-2.0.so.0
  #9  0xb7cd2120 in gtk_marshal_VOID__UINT_STRING () from /usr/lib/libgtk-x11-2.0.so.0
  #10 0xb77745d9 in g_cclosure_new_swap () from /usr/lib/libgobject-2.0.so.0
  #11 0xb777429b in g_closure_invoke () from /usr/lib/libgobject-2.0.so.0
  #12 0xb7788f7b in g_signal_has_handler_pending () from /usr/lib/libgobject-2.0.so.0
  #13 0xb778a0ad in g_signal_emit_valist () from /usr/lib/libgobject-2.0.so.0
  #14 0xb778a646 in g_signal_emit () from /usr/lib/libgobject-2.0.so.0
  #15 0xb7dd1e94 in gtk_widget_activate () from /usr/lib/libgtk-x11-2.0.so.0
  #16 0xb7cd0547 in gtk_propagate_event () from /usr/lib/libgtk-x11-2.0.so.0
  #17 0xb7cd07cf in gtk_main_do_event () from /usr/lib/libgtk-x11-2.0.so.0
  #18 0xb7b6b0e1 in gdk_event_get_graphics_expose () from /usr/lib/libgdk-x11-2.0.so.0
  #19 0xb76618e7 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
  #20 0xb7663285 in g_main_context_acquire () from /usr/lib/libglib-2.0.so.0
  #21 0xb76635aa in g_main_loop_run () from /usr/lib/libglib-2.0.so.0
  #22 0xb7ccfb23 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
  #23 0x0804a7cc in main (argc=1, argv=0xbfe07004) at main.c:96
  (gdb) 
  

  On top of the backtrace list are functions that were called last, so in the bottom of the list is the applications main() function and then the gtk_main() and so on.
• If you would now continue debugging with s (step) command you would end up looping the gtk main event loop but it is better to just give the c (continue) command:

  (gdb) c
  Continuing.
  

• Now the maemopad application is running and if you select the HELP menu item again the breakpoint is still set. So if you click HELP you would get again:

  Breakpoint 1, callback_help (action=0x805e768, data=0x8067170) at ui/callbacks.c:206
  206     {
  (gdb) 
  

• You can clear a specific breakpoint using the clear command. Lets remove the breakpoint in callback_help() function:

  (gdb) clear callback_help
  Deleted breakpoint 1 
  (gdb) cont
  Continuing.
  

• Because the breakpoint is now cleared you can now use the application normally under the Xephyr.

Debugging Hildon Desktop Plugins

This chapter explains how to debug Hildon Desktop plugins in maemo environment.

Desktop and Control Panel applications are both launched by maemo-launcher daemon. Their plugins are all shared libraries (.so files). How to write these plugins is explained in the maemo document How to write Hildon Desktop plugins for maemo.

Download and compile the example apps

For this example you need to first download, compile and install the hello-world-app package.

• Download the source package of hello-world-app:

  [sbox-SDK_X86: ~/src] > apt-get source hello-world-app
  Reading Package Lists... Done
  Building Dependency Tree... Done
  Need to get 320kB of source archives.
  Get:1 http://repository.maemo.org bora/free hello-world-app 0.4.1 (dsc) [317B]
  Get:2 http://repository.maemo.org bora/free hello-world-app 0.4.1 (tar) [319kB]
  Fetched 320kB in 1s (213kB/s)           
  dpkg-source: warning: extracting unsigned source package (./hello-world-app_0.4.1.dsc)
  dpkg-source: extracting hello-world-app in hello-world-app-0.4.1
  dpkg-source: unpacking hello-world-app_0.4.1.tar.gz
  
  

• Go to the sources directory ...

  [sbox-SDK_X86: ~/src] > cd hello-world-app-0.4.1
  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > 
  	
  

• ... and compile it like this:

  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > export DEB_BUILD_OPTIONS=debug,nostrip
  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > ./autogen.sh
  ... etc ...
  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > dpkg-buildpackage -rfakeroot
  dpkg-buildpackage: source package is hello-world-app
  dpkg-buildpackage: source version is 0.4.1
  dpkg-buildpackage: source changed by xxx yyy
  dpkg-buildpackage: host architecture i386
  dpkg-buildpackage: source version without epoch 0.4.1
  dpkg-checkbuilddeps: Using Scratchbox tools to satisfy builddeps
   fakeroot debian/rules clean
  dh_testdir
  dh_testroot
  rm -f build-stamp configure-stamp
  # Add here commands to clean up after the build process.
  /scratchbox/tools/bin/make clean
  
  ... snip, output from compilation ...
  
  dpkg-genchanges: including full source code in upload
  dpkg-buildpackage: full upload; Debian-native package (full source is included)
  
  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > 
  

• You should now have:

  [sbox-SDK_X86: ~/src/hello-world-app-0.4.1] > cd ..
  [sbox-SDK_X86: ~/src] > ls -lt 
  total 508
  -rw-rw-r--  1 user user    755 Mar 26 14:24 hello-world-app_0.4.1_i386.changes
  -rw-r--r--  1 user user  38672 Mar 26 14:24 hello-world-app_0.4.1_i386.deb
  drwxr-xr-x  6 user user   4096 Mar 26 14:23 hello-world-app-0.4.1
  -rw-rw-r--  1 user user    317 Mar 26 14:23 hello-world-app_0.4.1.dsc
  -rw-rw-r--  1 user user 318421 Mar 26 14:23 hello-world-app_0.4.1.tar.gz
  

• Install the newly compiled debug version of the package hello-world-app_0.4.1_i386.deb:

  [sbox-SDK_X86: ~/src] > fakeroot dpkg -i hello-world-app_0.4.1_i386.deb
  (Reading database ... 15186 files and directories currently installed.)
  Unpacking hello-world-app (from hello-world-app_0.4.1_i386.deb) ...
  ... snip ...
  
  

You should now have the Hildon desktop plugins example applications installed in your Scratchbox X86 target.

How to debug maemo-launcher started applications

Basically, you can debug these applications by:

• attaching to an already running process
• start the application using maemo-summoner

Attaching to maemo-launched application with gdb

With maemo-launched applications you need to give maemo-launcher binary to gdb and attach to the already running process.

[sbox-SDK_X86: ~/ ] af-sb-init.sh start
... snip ...
[sbox-SDK_X86: ~/ ] pidof maemo_af_desktop | cut -d' ' -f1
22961
#
# this would take the first (largest) PID value from the returned list. The number 22961 is just an example.
# smallest PID value is maemo-invoker which had requested maemo-launcher to start maemo_af_desktop.
#
[sbox-SDK_X86: ~/ ] gdb maemo-launcher
... snip ...
(gdb) attach 22961
Attaching to program: /targets/SDK_X86/usr/bin/maemo-launcher, process 22961
... snip...
(gdb)

You should now be able to debug the application normally with the gdb.

Starting maemo-launched application with maemo-summoner

Here we will start the Control Panel under (native) gdb and debug the newly installed Control Panel applet called hello-world-app. We will set the breakpoint to function hello_world_dialog_show(). Note the question about the "pending shared library load".

[sbox-SDK_X86: ~/ ] af-sb-init.sh start
... snip ...
[sbox-SDK_X86: ~/ ] gdb maemo-summoner
... snip ...
(gdb) br hello_world_dialog_show
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (hello_world_dialog_show) pending.
(gdb) run /usr/bin/controlpanel.launch
Starting program: /targets/SDK_X86/usr/bin/maemo-summoner /usr/bin/controlpanel.launch
... snip ...
	  

This should start the Control Panel in your Xephyr screen. When you click the hello world plugin in the Control Panel the execution should stop at the given breakpoint. You could try for example to get a backtrace:

Breakpoint 2, hello_world_dialog_show () at libhelloworld.c:96
96        GtkWidget *dialog = GTK_WIDGET(hello_world_dialog_new ());
(gdb) bt
#0  hello_world_dialog_show () at libhelloworld.c:96
#1  0xb70e1de7 in execute (osso=0x806bfd0, data=0x80dd820, user_activated=1) at hello-world-applet.c:38
#2  0xb7e284c5 in hcp_item_sort_func () from /usr/bin/controlpanel.launch
#3  0xb75e4ae1 in g_child_watch_add () from /usr/lib/libglib-2.0.so.0
#4  0xb75e18e7 in g_main_context_dispatch () from /usr/lib/libglib-2.0.so.0
#5  0xb75e3285 in g_main_context_acquire () from /usr/lib/libglib-2.0.so.0
#6  0xb75e35aa in g_main_loop_run () from /usr/lib/libglib-2.0.so.0
#7  0xb78ccb23 in gtk_main () from /usr/lib/libgtk-x11-2.0.so.0
#8  0xb7e26e0a in main () from /usr/bin/controlpanel.launch
#9  0x080488b5 in ?? ()
#10 0x00000001 in ?? ()
#11 0xbfce3e78 in ?? ()
#12 0x0804a050 in ?? ()
#13 0xb7f5188a in __do_global_ctors_aux () from /lib/libdl.so.2
#14 0xb7e462bb in __libc_start_main () from /lib/libc.so.6
#15 0x080486b1 in ?? ()

(gdb) list
91      }
92
93      void
94      hello_world_dialog_show ()
95      {
96        GtkWidget *dialog = GTK_WIDGET(hello_world_dialog_new ());
97        gtk_dialog_run (GTK_DIALOG(dialog));
98        gtk_widget_destroy (dialog);
99      }
100
(gdb) cont
Continuing.
	  

The question marks are shown because there are no debug symbols for maemo-summoner (we did not install any debug package for that). The backtrace tells us what functions were called before we reached the breakpoint at hello_world_dialog_show(). You could now debug the plugin normally with gdb.

Because the hello world plugin was compiled with -g option you can see the source code listing with the list command.

To do this same for desktop you need to kill maemo_af_desktop process first. You can kill the desktop with the command:

[sbox-SDK_X86: ~] > kill $(pidof maemo_af_desktop)
#
# after this you could start the maemo_af_desktop under <code>gdb</code> like this:
#
[sbox-SDK_X86: ~] > gdb maemo-summoner
... snip ...
(gdb) run /usr/bin/maemo_af_desktop.launch
... snip ...
	  

Running out-of-memory during debugging in device

If you run out of RAM memory during debugging in the device you have couple of options:

• Add (more) swap to the device using the Memory applet from Control Panel.

  If you should have enough memory but gdb is still abruptly terminated you could try setting it OOM-protected as root:

  Nokia-N800 # echo -17 > /proc/[PID of your gdb]/oom_adj
  	    
  

  By default processes have OOM (=Out-Of-Memory) adjustment value of zero. The value -17 disables kernel OOM killing for the given process. Note that as a result of this some other processes might be killed by kernel.

• Use gdbserver to debug

Notes about using gdbserver

Gdbserver is a debugging tool that you can start and run in the Internet Tablet device. You will then connect to this running instance of gdbserver program from your Linux PC with a gdb program. Gdbserver uses a lot less memory than a full scale gdb program and thus makes it possible to do debugging in devices that have a limited RAM memory, like PDA's and other gadgets.

The gdbserver does not care about symbols in the binary but instead the Linux PC side gdb expects to have a local copy of the binary being debugged so binaries in the device can be stripped.

For further information about using gdbserver, gdb and DDD please see:

• http://www.scratchbox.org/documentation/user/scratchbox-1.0/html/ideintegration.html#AEN76
• http://www.scratchbox.org/documentation/user/scratchbox-1.0/html/debugging.html

Strace debugger

Introduction to strace tool

Strace is a handy tool to see how your application is interacting with the operating system using the system calls. It displays to you what system calls are called and with what parameters. It also informs you of the return codes of the system calls. In addition, strace can attach to a running process making it suitable tool to debug what system calls the already running processes are calling.

Strace is especially helpful if you need to:

• Figure out why your program is failing to start or exits early
• Find what (configuration) files are used by the program
• Check is your program idle or wasting battery when it shouldn't do anything
• Know where a frozen program got stuck
• See what your program is communicating with the network

Installing the strace tool

You can install strace to the Internet Tablet and in Scratchbox you can use the Scratchbox provided strace. Using strace in X86 target is preferred for the reason that in ARMEL target strace will actually trace the system calls of the QEMU process (with which your arm binary is run).

To install the strace to the device just do:

Nokia-N800-01 ~# apt-get install strace
.... snip ...
	

Using the strace tool

These examples can be run both in Scratchbox X86 environment and in the Internet Tablet device itself.

Getting a system call trace from application

To get a simple list of the system calls the program is calling just run the strace by giving the path to the program as a parameter. Here we run the strace with the small gdb_example program we introduced before.

[sbox-SDK_X86: ~/src/testing/gdb_example] > strace ./gdb_example
execve("./gdb_example", ["./gdb_example"], [/* 41 vars */]) = 0
uname({sys="Linux", node="nema", ...})  = 0
brk(0)                                  = 0x804a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/sse2/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/sse2", 0xbf89fdac) = -1 ENOENT (No such file or directory)
open("/lib/tls/i686/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686", 0xbf89fdac)     = -1 ENOENT (No such file or directory)
open("/lib/tls/sse2/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/sse2", 0xbf89fdac)     = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0xbf89fdac)          = -1 ENOENT (No such file or directory)
open("/lib/i686/sse2/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/i686/sse2", 0xbf89fdac)    = -1 ENOENT (No such file or directory)
open("/lib/i686/libc.so.6", O_RDONLY)   = -1 ENOENT (No such file or directory)
stat64("/lib/i686", 0xbf89fdac)         = -1 ENOENT (No such file or directory)
open("/lib/sse2/libc.so.6", O_RDONLY)   = -1 ENOENT (No such file or directory)
stat64("/lib/sse2", 0xbf89fdac)         = -1 ENOENT (No such file or directory)
Open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200S\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1164796, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40018000
old_mmap(NULL, 1174836, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40019000
old_mmap(0x40132000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x118000) = 0x40132000
old_mmap(0x40136000, 7476, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40136000
close(3)                                = 0
mprotect(0x40132000, 4096, PROT_READ)   = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 4), ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40138000
write(1, "Now in main().\n", 15Now in main().
)        = 15
write(1, "Now in example_1(1) function.\n", 30Now in example_1(1) function.
) = 30
write(1, "Now in example_2(parameter here)"..., 43Now in example_2(parameter here) function.
) = 43
write(1, "Now in example_3()\n", 19Now in example_3()
)    = 19
munmap(0x40138000, 4096)                = 0
exit_group(0)                           = ?
[sbox-SDK_X86: ~/src/testing/gdb_example] > 
	  

Essentially one line in the output refers to one system call made to the Linux kernel. Every line has the name of the system call, then the parameters to the call and finally the return value of the system call.

For example, the line:

write(1, "Now in main().\n", 15Now in main().
)        = 15
	    

tells us that the system call that was called from gdb_example was write() with parameters 1,"Now in main().\n", 15 and then the running gdb_example program dumps the "Now in main()." text in the middle of the output so that the actual return code is display in the end of the line =15.

System calls that fail often return (but not always) a return code of -1.

An example of failed stat64 call would be:

stat64("/lib/sse2", 0xbf89fdac)         = -1 ENOENT (No such file or directory)
	    

Here the stat64 return value of -1 is displayed together with the symbolic name of ENOENT. This tells immediately that the program tried to access /lib/sse2 but failed for the reason that the file does not exists.

Studying the output from strace and comparing it to the source file tells you that forexample the gdb_example programs printf() functions are each calling the write() system call. In addition you can see all the open() and stat64() and other system calls in the list.

Attaching to a running process with strace

You can attach to a running process with strace. This is helpful especially if you are debugging a daemon type of an application and want to see what system calls the daemon is calling without re-starting the daemon.

In this example we first start the top application inside scratchbox and leave it running in one terminal and then open another terminal and attach to the top application.

• Start the top application inside one terminal in Scratchbox X86 target:

  [sbox-SDK_X86: ~] > top
  
  ## the screen clears and top starts to loop...
  
  top - 14:49:35 up 17 days, 23:38,  0 users,  load average: 0.07, 0.02, 0.00
  Tasks: 115 total,   2 running, 113 sleeping,   0 stopped,   0 zombie
  Cpu(s):  0.0% us,  0.0% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
  Mem:    775176k total,   754708k used,    20468k free,   126608k buffers
  Swap:  1622524k total,    19716k used,  1602808k free,   413752k cached
  
    PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                
  12807 user      16   0  2184 1172  888 R 79.0  0.2   0:00.09 top
      1 root      16   0  1632  536  448 S  0.0  0.1   0:01.56 init
  2 root      34  19     0    0    0 R  0.0  0.0   0:00.08 ksoftirqd/0
  3 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 watchdog/0 
  4 root      10  -5     0    0    0 S  0.0  0.0   0:00.18 events/0 
  5 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 khelper
  6 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
  8 root      10  -5     0    0    0 S  0.0  0.0   0:00.11 kblockd/0
  9 root      10  -5     0    0    0 S  0.0  0.0   0:00.05 kacpid
  ... etc ...
  		
  
  

• Next, start another terminal window and login to scratchbox and run:

  [sbox-SDK_X86: ~] > pidof top
  12807
  [sbox-SDK_X86: ~] > strace -p 12807
  		
  

  With -p flag we can attach strace to a running process.

• With the pidof command we can easily find the PID of a running program. In this example it is 12807 (in your machine the number will be different). The output of the strace would look similar to this:

  Process 12807 attached - interrupt to quit
  select(1, [0], NULL, NULL, {2, 536000}) = 0 (Timeout)
  fcntl64(0, F_SETFL, O_RDWR|O_NONBLOCK|O_LARGEFILE) = 0
  read(0, 0xbfca532f, 1)                  = -1 EAGAIN (Resource temporarily unavailable)
  ioctl(0, TCFLSH, 0)                     = 0
  fcntl64(0, F_SETFL, O_RDWR|O_LARGEFILE) = 0
  gettimeofday({1174308898, 584273}, {4294967176, 0}) = 0
  open("/proc", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 7
  fstat64(7, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
  fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
  getdents64(7, /* 168 entries */, 8192)  = 4552
  stat64("/proc/1", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
  open("/proc/1/stat", O_RDONLY)          = 8
  read(8, "1 (init) S 0 1 1 0 -1 8388864 19"..., 1023) = 192
  close(8)                                = 0
  open("/proc/1/statm", O_RDONLY)         = 8
  read(8, "408 134 112 14 0 61 0\n", 1023) = 22
  close(8)                                = 0
  stat64("/proc/2", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
  open("/proc/2/stat", O_RDONLY)          = 8
  read(8, "2 (ksoftirqd/0) S 1 1 1 0 -1 410"..., 1023) = 131
  close(8)                                = 0
  open("/proc/2/statm", O_RDONLY)         = 8
  read(8, "0 0 0 0 0 0 0\n", 1023)        = 14
  close(8)                                = 0
  stat64("/proc/3", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
  open("/proc/3/stat", O_RDONLY)          = 8
  read(8, "3 (watchdog/0) S 1 1 1 0 -1 4128"..., 1023) = 132
  close(8)                                = 0
  
  ... long list of output ...
  		
  
  

  Studying the output of the strace we can see what system calls the top program is calling during its main loop. In this case, top is opening a lot of files under the /proc/ file system to get information about the running processes in the system.

How to trace all applications running in the device

Occasionally you may want to trace all running applications in the device. For example you may want to know what process accesses some specific file, but it does not keep it open all the time (if it is kept open all the time you can find it from /proc/*/fd/ or with lsof).

In this example we trace open() system calls from all running programs in the device started after the DSME process.

	    Nokia-N800-01 ~# strace -e trace=open \
	      $(cd /proc/; dsme=$(pidof dsme|cut -d' ' -f1); \
	      for pid in [0-9]*; do \
	         if [ $pid -gt $dsme ] && [ $pid != $$ ]; then \
	             echo "-p $pid"; \
	         fi; \
	      done) 
	  

Notice that the above command does not start to trace the DSME. Accidentally suspending DSME would cause the device to reboot because DSME keeps the device HW watchdog refreshed.

The output of the command would look similar to this:

Nokia-N800-01:~# strace -e trace=open \
>       $(cd /proc/; dsme=$(pidof dsme|cut -d' ' -f1); \
>       for pid in [0-9]*; do \
>          if [ $pid -gt $dsme ] && [ $pid != $$ ]; then \
>              echo "-p $pid"; \

>          fi; \
>       done) 
Process 1019 attached - interrupt to quit
Process 1036 attached - interrupt to quit
Process 1039 attached - interrupt to quit
...snip...
Process 13559 attached - interrupt to quit
attach: ptrace(PTRACE_ATTACH, ...): No such process
Process 332 attached - interrupt to quit
Process 3606 attached - interrupt to quit
attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted
Process 434 attached - interrupt to quit
Process 682 attached - interrupt to quit
...snip...
Process 332 detached
[pid   762] open("/dev/dspctl/ctl", O_RDWR) = 6
[pid   762] open("/dev/dspctl/ctl", O_RDWR) = 6
[pid   762] open("/dev/dspctl/ctl", O_RDWR) = 6
[pid   709] --- SIGALRM (Alarm clock) @ 0 (0) ---
[pid   709] --- SIGALRM (Alarm clock) @ 0 (0) ---
[pid   762] open("/dev/dspctl/ctl", O_RDWR) = 6
[pid   709] --- SIGALRM (Alarm clock) @ 0 (0) ---
... etc ...
	  

From the output you can see which processes strace attaches to and what are the files those processes are opening.

Other strace options

Here is a list of useful strace options and how to use them.

• -f fork

  The -f fork option tells strace to follow the execution to the forked process from the calling process. By default strace does not follow forks.

• -e expr

  The -e option is handy if you want to limit the scope of the output. For example doing:

  [sbox-SDK_X86: ~/src/testing/gdb_example] > strace -e trace=network ./gdb_example
  		
  

  would only output those calls that are network related. In this example there would be no other output than the normal output of the program.

  But if you try:

  [sbox-SDK_X86: ~/src/testing/gdb_example] > strace -e trace=file ./gdb_example
  		
  

  You would only see what system calls are called that are related to accessing files.

Valgrind debugger

Introduction to Valgrind tool

Valgrind is a CPU simulator with different debugging and analyzing plugins. The Valgrind plugins are:

• memcheck
  This plugin tool is used to debug memory leaks and deallocation errors in applications. In this example we focus mainly on this.
• massif
  This plugin produces PostScript graph of process memory usage as a function of time. This also produces ASCII or HTML report of allocation backtraces.
• callgrind
  This can be used for profiling performance and getting call traces from programs. These can be further visualized with Kcachegrind tool.
• helgrind
  This helps you in finding race conditions in threaded programs. This does not work in maemo Valgrind. It works only with Valgrind 2.2.

Valgrind has many options but only the basic ones are covered here with a small example. The link to the full Valgrind manual is given in the end of this chapter.

Installing the Valgrind tool

Installing Valgrind is simple. Login to Scratchbox and run the following commands:

• Get Valgrind from the repository:

  
  [sbox-SDK_ARMEL: ~] > sb-conf select SDK_X86
  [sbox-SDK_X86: ~] > apt-get install valgrind
  Reading Package Lists... Done
  Building Dependency Tree... Done
  The following extra packages will be installed:
    libc6-dbg
  Suggested packages:
    kcachegrind alleyoop
  The following NEW packages will be installed:
    libc6-dbg valgrind
  0 upgraded, 2 newly installed, 0 to remove and 9 not upgraded.
  Need to get 16.8MB of archives.
  After unpacking 43.7MB of additional disk space will be used.
  Do you want to continue? [Y/n] 
  Get:1 http://repository.maemo.org bora/free libc6-dbg 2.3.5cs2005q3.2-5.osso12 [4807kB]
  Get:2 http://repository.maemo.org bora/free valgrind 1:3.2.0-2.osso2 [12.0MB]                                                                      
  Fetched 16.8MB in 45s (370kB/s)                                                                                                                    
  Selecting previously deselected package libc6-dbg.
  (Reading database ... 14955 files and directories currently installed.)
  Unpacking libc6-dbg (from .../libc6-dbg_2.3.5cs2005q3.2-5.osso12_i386.deb) ...
  Selecting previously deselected package valgrind.
  Unpacking valgrind (from .../valgrind_1%3a3.2.0-2.osso2_i386.deb) ...
  Setting up libc6-dbg (2.3.5cs2005q3.2-5.osso12) ...
  
  Setting up valgrind (3.2.0-2.osso2) ...
  
  [sbox-SDK_X86: ~] > 
  	
  

  The maemo Valgrind version depends on the libc6-dbg. On the desktop Linux some of the debug symbols are included in the libc6 library itself. If the debug symbols are missing from the libraries then Valgrind can not match the error suppressions to the internal library functions. In the maemo libc6 case it would show lots of errors for the dynamic linker.
  If you use a non-maemo version of Valgrind you need to set the following environment variable before valgrinding programs using Glib:
  [sbox-SDK_X86: ~] > export G_SLICE="always-malloc"
  Without this Valgrind reports bogus leaks from Glib.
  

• Next, get the two small example applications written in C-language to demonstrate the basic usage of valgrind tool. You can download these from maemo.org:

  [sbox-SDK_X86: ~] > mkdir src
  [sbox-SDK_X86: ~] > cd src
  [sbox-SDK_X86: ~/src] > 
  

  You can download the small valgrind example applications from stage.maemo.org. See links in the end of this document.

  After downloading just copy the valgrind_example.tar.gz file to your ~/src/ directory and do:

  [sbox-SDK_X86: ~/src] > tar xvzf valgrind_example.tar.gz
  valgrind_example/
  valgrind_example/valgrind_example.c
  valgrind_example/valgrind_example2.c
  
  [sbox-SDK_X86: ~/src] > cd valgrind_example
  [sbox-SDK_X86: ~/src/valgrind_example] > 
  
  

• Next compile the two small example applications like this:

  [sbox-SDK_X86: ~/src/valgrind_example] > gcc valgrind_example.c -o valgrind_example -g 
  [sbox-SDK_X86: ~/src/valgrind_example] > gcc valgrind_example2.c -o valgrind_example2 -g 
    
  

You are now ready to move the next chapter.

Using the Valgrind memory debugger tool

After compiling the small example application run it under valgrind with a command:

[sbox-SDK_X86: ~/src/valgrind_example] > valgrind --tool=memcheck --leak-check=yes --show-reachable=yes \
                                 --num-callers=20 --track-fds=yes ./valgrind_example
	

Explaining the parameters and their meaning:

• --tool=memcheck

  Defines what tool Valgrind should use. In this example we use the memory checker tool.

• --leak-check=yes

  With this option Valgrind checks your code for potential memory leaks.

• --show-reachable=yes

  This option creates a stack trace of the creation of the reachable but unfreed memory when the program exits.

• --num-callers=20

  With this option you can change the number of function call levels that Valgrind displays to you. The default value is 12. Giving higher number takes a bit more memory. You might want to tune this option because for example the gtk callstack can be more than 100 items.

• --track-fds=yes

  Track-fds means Track FileDescriptors. If your application is opening and closing files with fopen() or open() this will report which files are not closed.

The output of the example application should be like:

==23913== Memcheck, a memory error detector.
==23913== Copyright (C) 2002-2006, and GNU GPL'd, by Julian Seward et al.
==23913== Using LibVEX rev 1606, a library for dynamic binary translation.
==23913== Copyright (C) 2004-2006, and GNU GPL'd, by OpenWorks LLP.
==23913== Using valgrind-3.2.0-Debian, a dynamic binary instrumentation framework.
==23913== Copyright (C) 2000-2006, and GNU GPL'd, by Julian Seward et al.
==23913== For more details, rerun with: -v
==23913== 
==23913== 
==23913== FILE DESCRIPTORS: 3 open at exit.
==23913== Open file descriptor 2: /dev/pts/4
==23913==    <inherited from parent>
==23913== 
==23913== Open file descriptor 1: /dev/pts/4
==23913==    <inherited from parent>
==23913== 
==23913== Open file descriptor 0: /dev/pts/4
==23913==    <inherited from parent>
==23913== 
==23913== 
==23913== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 11 from 1)
==23913== malloc/free: in use at exit: 40 bytes in 2 blocks.
==23913== malloc/free: 3 allocs, 1 frees, 60 bytes allocated.
==23913== For counts of detected errors, rerun with: -v
==23913== searching for pointers to 2 not-freed blocks.
==23913== checked 51,552 bytes.
==23913== 
==23913== 10 bytes in 1 blocks are definitely lost in loss record 1 of 2
==23913==    at 0x401C4A6: malloc (vg_replace_malloc.c:149)
==23913==    by 0x804841B: main (valgrind_example.c:19)
==23913== 
==23913== 
==23913== 30 bytes in 1 blocks are definitely lost in loss record 2 of 2
==23913==    at 0x401C4A6: malloc (vg_replace_malloc.c:149)
==23913==    by 0x8048482: main (valgrind_example.c:41)
==23913== 
==23913== LEAK SUMMARY:
==23913==    definitely lost: 40 bytes in 2 blocks.
==23913==      possibly lost: 0 bytes in 0 blocks.
==23913==    still reachable: 0 bytes in 0 blocks.
==23913==         suppressed: 0 bytes in 0 blocks.
[sbox-SDK_X86: ~/src/valgrind_example] > 
	

The output of the Valgrind tells us that there were 40 unallocated bytes for the application ("definitely lost"). This means that the example application is leaking memory which it cannot anymore free and you should study the code closely.

Valgrind also tells us that in what lines in your code these allocations that are not freed are done. In this example these are happening at lines 41 and 19.

Official Valgrind Manual

Valgrind.org has an official Valgrind manual available in the Internet that explains the many options and debugging practices that Valgrind can support. For full coverage of Valgrind see:

http://valgrind.org/docs/manual/manual.html

Code Examples

Here are the two example code packages that were used in this text.

• gdb_example.tar.gz
• valgrind_example.tar.gz