ImprovementIdeasForOssoEmail

  1. About
  2. A proposed solution
    1. Custom model
    2. Proxy instances
  3. Thoughts and concerns
  4. Some violations of model code in the view
  5. So what should happen
  6. Other ideas
  7. Improvement in e-mail Retrieval

About

https://stage.maemo.org/svn/maemo/projects/email/osso-email (use guest/guest as u/p)

In the Model View Controller paradigm, you typically create a rather dumb view that's only responsible for viewing the data. Not managing it.

The managing should happen in both the model and the controller code.

The view becomes an observer of the controller and/or model.

In principle it's not a necessity to have all the memory headers of all e-mails in memory of the device at all times. Yet, due it's current design, this is exactly what osso-email does. Another component (since it's transferring the headers over d-bus) might even be holding yet another copy of all the headers (in another format). Probably it's freeing that, but it is (however) performing things in order to make it possible to pass the data.

osso_email_xml_unpack_msgheader_list both allocates the data and parses the XML data? So the data was once parsed into a xmlDocPtr as it's again stored (as a copy) in a GSList of MsgHeader instances as it's going to be pointer-copied to the GtkTreeModel (as the standard GtkListStore is being used). That's three large list of pointers and two lists with the data (both, copies).

The GtkTreeModel holds only 'references' to the char pointers (so, that's more or less okay). But why is it transferred as XML (or whatever format)? This ADDS an extra parsing step at the ossomail process, AND a xml writing step at the enengine process. On a huge server pc, this is cheap. On the 770, it's extremely expensive for no reason whatsoever (d-bus can transfer structs without having to pack 'em).

Different question, how is ui_dbus_get_msgheader_list ever going to return something different from NULL?

A proposed solution

Custom model

Sample: https://svn.cronos.be/svn/custom-treemodel-demo/trunk/

If a custom model was build, rather than using GtkTreeModel, it could read from a on-disk cache the most likely to be 'viewed' headers. The header-view displays only +- 50 headers. Let's say one loads 25 above and 25 beneath the 50 that are visible. The view will 'request' the ones when the user starts scrolling, sorting, etcetera.

Or when a sorting change happens, the view will 'request' the 100 headers that are 'now' visible.

This would, indeed, make a lot operations slower. But the most common ones, would be 'a lot' faster.

Take a look at the ETable TreeView implementation in libgal. This implementation is being used by Evolution (check below for links and references).

Proxy instances

A proxy class is a class that delegates the 'real work' to a 'real subject'. Lets say the subject is, in our case, a MsgHeader. We'd create an interface IMsgHeader that plays the role of a contract for the components that will depend on a message header (for example the View and the Controller).

Both the real subject (MsgHeader) and the proxy MsgHeaderPxy implement the IMsgHeader interface. The proxy, however, doesn't really contain the data nor implementations. But when it's first needed, it'll create a real MsgHeader instance (for example an aggregation or a factory creation) and will perform the actions it needs to fullfil on that instance.

So in a way are proxy classes 'sleeping' IMsgHeader implementations. When they are needed, they wake up (and start consuming memory).

Most systems (like most likely also the N770) can easily handle 10.000ths of such proxy instances. Often then can't handle 10.000ths of instances of real such subjects.

For this idea to succeed, the GtkTreeModel needs to support calling "getter" functions rather than using a gchar pointer for accessing the data itself.

A sample: http://www.pvanhoof.be/wiki/index.php/Smart_ways_of_using_GtkTreeView

Also note that a more complete sample that also uses a custom treemodel can be found here: https://svn.cronos.be/svn/custom-treemodel-demo/trunk/

#include <gtk/gtk.h>

static int ELEMENT_COUNT = 10000;
static int instantiations = 0;

static GtkWidget *window = NULL;

typedef struct _IMsgHeader IMsgHeader;
typedef struct _MsgHeader MsgHeader;
typedef struct _MsgHeaderPrx MsgHeaderPrx;

struct _IMsgHeader {
   const gchar* (*get_from_func) (IMsgHeader *this);
   const gint   (*get_id_func)   (IMsgHeader *this);

   gint id;
};

struct _MsgHeader {
   IMsgHeader parent;
   gchar *from;
};

struct _MsgHeaderPrx {
   IMsgHeader parent;
   MsgHeader *real;
};


/* IMsgHeader (late binding like) impl */
const gint
imsg_header_get_id (IMsgHeader *this)
{
    return this->get_id_func (this);
}

const gchar* 
imsg_header_get_from (IMsgHeader *this)
{
    return this->get_from_func (this);
}


/* Real subject impl */
MsgHeader* 
msg_header_new (gint id)
{
    MsgHeader *header = g_new0(MsgHeader, 1);

        g_print ("Instantiation for id=%d (%d passed)\n!!",
         id, instantiations);

    instantiations++;

    header->parent.id = id;
    header->from = g_strdup ("A real IMsgHeader subject!");

    return header;
}


/* Proxy impl */
const gchar*
msg_header_proxy_get_from (IMsgHeader *this_i)
{
   MsgHeaderPrx *this = (MsgHeaderPrx*)this_i;

   if (this->real == NULL)
       this->real = msg_header_new (this->parent.id);

   return this->real->from;
}

const gint
msg_header_proxy_get_id (IMsgHeader *this_i)
{
   MsgHeaderPrx *this = (MsgHeaderPrx*)this_i;

   if (this->real == NULL)
       this->real = msg_header_new (this->parent.id);

   return this->real->parent.id;
}


MsgHeaderPrx*
msg_header_proxy_new (gint id)
{
   MsgHeaderPrx *header = g_new0(MsgHeaderPrx, 1);

   header->parent.id = id;
   header->parent.get_from_func = msg_header_proxy_get_from;
   header->parent.get_id_func = msg_header_proxy_get_id;

   header->real = NULL;

   return header;
}


enum
{
   COLUMN_NUMBER,
   COLUMN_HEADER, 
   NUM_COLUMNS
};


static void
msg_header_treeview_destroy_model_item (gpointer data)
{
        g_print ("Destroy\n");
}

static void
msg_header_treeview_get_model_item (GtkTreeViewColumn *tree_column,
                                   GtkCellRenderer   *cell,
                                   GtkTreeModel      *tree_model,
                                   GtkTreeIter       *iter,
                                   gpointer           data)
{
        gint number;
        IMsgHeader *header;
    GValue val = {0,};
    
        /* This function can call the getter on the item in the model.
         * Such an item can be a proxy class (IMsgHeaderPrx), triggering
         * this getter causes an instantiation (or getting it from a
         * factory with a cache) of a real subject (a real MsgHeader).
         *
         * The proxy now reads the value from the real subject and
         * returns it to fullfill the contract of the interface IMsgHeader.
         * /

        gtk_tree_model_get (tree_model, iter, COLUMN_HEADER, &header, -1);
        g_print ("from=%s, %d\n", imsg_header_get_from (header), 
            imsg_header_get_id (header));

    g_value_init (&val, G_TYPE_STRING);
    g_value_set_string (&val,  imsg_header_get_from (header));
    g_object_set_property (G_OBJECT (cell), "text", &val);

    g_value_unset (&val);
}

static GtkTreeModel *
create_model (void)
{
  gint i = 0;
  GtkListStore *store;
  GtkTreeIter iter;

  store = gtk_list_store_new (NUM_COLUMNS,
                              G_TYPE_INT,
                              G_TYPE_POINTER);


  for (i = 0; i < ELEMENT_COUNT; i++)
  {

      MsgHeaderPrx *header = msg_header_proxy_new (i);

      gtk_list_store_append (store, &iter);
      gtk_list_store_set (store, &iter,
              COLUMN_NUMBER, header->parent.id,
                          COLUMN_HEADER, header, -1);
  }

  return GTK_TREE_MODEL (store);
}


static void
add_columns (GtkTreeView *treeview)
{
  GtkCellRenderer *renderer;
  GtkTreeViewColumn *column;
  GtkTreeModel *model = gtk_tree_view_get_model (treeview);

  renderer = gtk_cell_renderer_text_new ();
  column = gtk_tree_view_column_new_with_attributes ("Number",
                                                     renderer,
                                                     "text",
                                                     COLUMN_NUMBER,
                                                     NULL);
  gtk_tree_view_column_set_sort_column_id (column, COLUMN_NUMBER);
  gtk_tree_view_column_set_sizing (column, GTK_TREE_VIEW_COLUMN_FIXED);

  gtk_tree_view_column_set_fixed_width (column, 200);

  gtk_tree_view_column_set_cell_data_func (column, renderer, 
    msg_header_treeview_get_model_item, NULL,
    msg_header_treeview_destroy_model_item);

  gtk_tree_view_append_column (treeview, column);

}

GtkWidget *
do_list_store (void)
{
  if (!window)
    {
      GtkWidget *vbox;
      GtkWidget *label;
      GtkWidget *sw;
      GtkTreeModel *model;
      GtkWidget *treeview;

      /* create window, etc */
      window = gtk_window_new (GTK_WINDOW_TOPLEVEL);
      gtk_window_set_title (GTK_WINDOW (window), "GtkListStore demo");

      g_signal_connect (window, "destroy",
                        G_CALLBACK (gtk_widget_destroyed), &window);
      gtk_container_set_border_width (GTK_CONTAINER (window), 8);

      vbox = gtk_vbox_new (FALSE, 8);
      gtk_container_add (GTK_CONTAINER (window), vbox);

      label = gtk_label_new ("Bla Bla Bla");
      gtk_box_pack_start (GTK_BOX (vbox), label, FALSE, FALSE, 0);

      sw = gtk_scrolled_window_new (NULL, NULL);
      gtk_scrolled_window_set_shadow_type (GTK_SCROLLED_WINDOW (sw),
                                           GTK_SHADOW_ETCHED_IN);
      gtk_scrolled_window_set_policy (GTK_SCROLLED_WINDOW (sw),
                                      GTK_POLICY_NEVER,
                                      GTK_POLICY_AUTOMATIC);
      gtk_box_pack_start (GTK_BOX (vbox), sw, TRUE, TRUE, 0);


      g_print ("Create model\n");
      /* create tree model */
      model = create_model ();

      g_print ("Create treeview\n");

      /* create tree view */
      treeview = gtk_tree_view_new_with_model (model);
      gtk_tree_view_set_rules_hint (GTK_TREE_VIEW (treeview), TRUE);

      gtk_tree_view_set_fixed_height_mode (GTK_TREE_VIEW(treeview), TRUE);

      g_object_unref (model);

      gtk_container_add (GTK_CONTAINER (sw), treeview);

      /* add columns to the tree view */
      add_columns (GTK_TREE_VIEW (treeview));

      /* finish & show */
      gtk_window_set_default_size (GTK_WINDOW (window), 280, 250);
    }

  if (!GTK_WIDGET_VISIBLE (window))
    gtk_widget_show_all (window);
  else
    {
      gtk_widget_destroy (window);
      window = NULL;
    }

  return window;
}

int main (int argc, char **argv)
{
        GtkWindow *win;
        gtk_init (&argc, &argv);

        if (argc > 1)
        {
                ELEMENT_COUNT = atoi (argv[1]);
        }

        win = GTK_WINDOW(do_list_store ());
        gtk_widget_show_all (GTK_WIDGET(win));

        gtk_main();

}

Something like this could be used to kill the instances of invisible rows. For example putting it in the msg_header_treeview_get_model_item will do the trick. Note that this WILL slow down the treeview, but WILL make sure that only the instances that are really required are alive in the memory. Do use a memory-pool for the real subjects if you intend to use this. Else you're process will suffer from immense memory fragmentation.

A much better technique would be implementing a custom GtkTreeModel(IfAce) and implementing the unref and ref methods for this.


#define MSG_HEADER_PROXY_HAS_CACHE(o)   ((o)->real != NULL)
void
msg_header_proxy_uncache (MsgHeaderPrx *this)
{
    if (MSG_HEADER_PROXY_HAS_CACHE(this))
    {
        instances--; 
        g_print ("Free, %d alloctions\n", instances);
        g_free (this->real->from);
        g_free (this->real);
        this->real = NULL;
    }
}

static void 
kill_unvisible (GtkTreeView *view, GtkTreeModel *tree_model)
{
    GtkTreePath *vis_start=NULL, *vis_end=NULL;
    gboolean valid = FALSE, go = FALSE;

    /* Search for unvisible rows and release their data */
    go = gtk_tree_view_get_visible_range (view, &vis_start, &vis_end);
    if (go && vis_start != NULL && vis_end != NULL) 
    {
        GtkTreeIter iter_start, iter_end, iter;

        gtk_tree_model_get_iter (tree_model, &iter_start, vis_start);
        gtk_tree_model_get_iter (tree_model, &iter_end, vis_end);

        gtk_tree_path_free (vis_start);
        gtk_tree_path_free (vis_end);

        valid = gtk_tree_model_get_iter_first (tree_model, &iter);

        while (valid)
        {
            if (((&iter)->user_data > (&iter_end)->user_data) ||
                ((&iter)->user_data < (&iter_start)->user_data)) 
            {
                IMsgHeader *header = NULL;
                gtk_tree_model_get (tree_model, &iter, COLUMN_HEADER, &header, -1);
                if (header != NULL) 
                {
                    MsgHeaderPrx *prx = (MsgHeaderPrx*)header;
                    if (MSG_HEADER_PROXY_HAS_CACHE(prx))
                        msg_header_proxy_uncache (prx);
                }
            }
            valid = gtk_tree_model_iter_next (tree_model, &iter);
        }
    }
}

Try the same sample without the fixed-width properties of the columns and without the fixed-row-height properties of the treeview.

The result will be that in the background, a lot rows will be processed.

This is something you don't want to let happen on the Nokia 770, as this would take enormous amounts of (mainly) I/O and CPU processing time in case the instances in de list model are proxy classes and/or in case you've created a custom model with 'lazy' getters.

Thoughts and concerns

The TreeView implementation shouldn't accidently request the full list of items from the model. It should at all times only request the items that are visible.

It would be nice to have control over this behaviour. If for example it would be possible to also load the 25 items above and the 25 beneath the upper and lower visible item, it would make it more easy to make the view look faster.

If the view could request getting more items in a worker thread, it would also be nice. Perhaps even better would be loading the items non-blocking using a select-loop (async posix io operations).

Some violations of model code in the view

ui_get_message_display_time, ui_free_msg_header, ui_get_message_attachment_status

They should be in msg_header.c and msg_header.h and shouldn't be called ui_* and should use the model class rather than reading it from the view: standard model view controller paradigm.

This is where loading a massive amount of header data happens:

ui_populate_folder, ui_dbus_populate_folder, ui_add_listview_header.

So what should happen

Separate the list model, controller and view.

The list model is a model that contains a 'fake' list of MsgHeader instances. This means that it returns a certain length (the amount of available items on disk, the engine can populate that). The getter will check whether the item is in memory-cache and if not will cache it by getting it from the disk store. The setter will write to disk store and will invalidate the cache if it was cached OR will instruct the 'engine' to do this as a delegate for this operation. The setters also triggers the observers's update method about this event.

The list model should be an implementation of the GtkTreeModel if you'd want to use the GtkTreeView. Else creating a new treeview widget is a possible solution (or reuse existing ones like the one being used by Evolution).

The controller is responsible for dealing with operations the view in the current design performs on the model.

The view is 'only' responsible for viewing and for requesting items from the model. So the view 'requests' 25 items above the upper viewable item, 50 viewable items and 25 items below the lower viewable item. On events like scrolling and caching the sort order, the view will do this.

A GtkTreeView that uses a GtkTreeModel already does it's stuff like this (not sure about the 25 above and 25 below, but also that can easily be predicted by just the model which you're going to implement-- but it's more easy to predict it in the view, of course).

It will now also be more easy to integrate a "search" capability in the view. In this case will the view pass a query to the model. Other than the UI component to define that query, the view doesn't change a single line of it's code. Searching is implemented in the model, not the view. Dependency injection is, btw, a nice technique for getting such query- able lists as datasource in a model view controller paradigm.

Other ideas

Improvement in e-mail Retrieval

GNUGotMail for PalmOS from http://rallypilot.sourceforge.net/ has nice simple and useful features:

  • Retrieves from the last email reads
  • Specify Maximum of lines to be retrieves from an email.

From GNUGotMail site, "GNUGotMail is that the last message number is kept in the POP Preferences so that the next time you check your mail it will continue at that point. This is useful if you are checking your mail while away from your computer and you don't want to delete your mail from the POP.

If you specify a 'max number of lines' it will use the TOP command rather than the RETR command, which can shorten your session time as well as keep your Inbox messages from being too big. Since the TOP command counts "lines" it is not as accurate as a byte-count because lines may be up to 1024 bytes long according to the POP RFC. Still, it is more efficient than other truncation methods because you do not have to receive the entire message as you would with the RETR command."