How to use the Camera API
This document explains how to use the Camera API to access the camera hardware that is present in some models of Nokia Internet Tablets.
This document has been reviewed for maemo 3.x.
Camera hardware and Linux
The Linux operating system supports live video/audio hardware, like webcams, TV tuners, video capture cards, FM radio tuners, video output devices etc. The primary API for the applications to have access to those devices is Video4Linux.
Video4Linux is a kernel API, so there must be kernel drivers for each supported device. At the user level, device access is standartized via device files. In the case of video capture devices like cameras, which are the focus of this document, the files would be /dev/video0, /dev/video1... as many as there are devices connected.
Data exchanged between device file and user-level application has a standartized format for each device class. This allows the application to be instantly compatible with every video capture device that has a driver for Linux.
The built-in camera present in some Nokia Internet Tablet devices is compatible with Video-4-Linux version 2 API http://www.thedirks.org/v4l2/. In principle, any application compatible with this API is easily portable to the maemo platform.
Since the maemo Platform delegates all multimedia handling to the GStreamer framework, applications that need access to the built-in camera should employ GStreamer for this instead of directly acessing Video4Linux devices, via the v4l2src GStreamer module.
Thanks to the GStreamer's flexibility, the developer can fully test any given application in a regular desktop PC with a connected webcam, and then final test it in the Internet Tablet itself, without a single change in the source code since GStreamer refers to modules as text names.
One important note about the camera in the Tablet is that only one application can use it at a time. So, while you are using the camera in your application, other tasks that could possibly make use of it (e.g. a videocall) will be blocked.
The best demonstrate how the camera manipulation is done two example applications are provided and discussed, one in C and another in Python.
Camera manipulation in C language
The C application allows to use the Internet Tablet as a "mirror" (i.e. showing the camera input in the screen), as well as allows to take pictures that are saved as JPEG files (which illustrates the video frame buffer manipulation).
This first part is the boilerplate code necessary for every C source file. It also defines some constant and functions and also the AppData-structure which is used to pass around "app-wide" data.
#include <stdlib.h>
#include <string.h>
#include <gtk/gtk.h>
#include <gst/gst.h>
#include <gst/interfaces/xoverlay.h>
#include <hildon-widgets/hildon-banner.h>
#include <hildon-widgets/hildon-program.h>
#include "example_common.h"
/* Define save folder and output picture name.
* If SAVE_FOLDER_DEFAULT is NULL, pictures are
* written to the default tmp-directory */
#define SAVE_FOLDER_DEFAULT NULL
#define PHOTO_NAME_DEFAULT "Picture"
#define PHOTO_NAME_SUFFIX_DEFAULT ".jpg"
/* Define sources and sinks according to
* running environment
* NOTE: If you want to run the application
* in ARM scratchbox, you have to change these*/
#ifdef __arm__
/* The device by default supports only
* vl4l2src for camera and xvimagesink
* for screen */
#define VIDEO_SRC "v4l2src"
#define VIDEO_SINK "xvimagesink"
#else
/* These are for the X86 SDK. Xephyr doesn't
* support XVideo extension, so the application
* must use ximagesink. The video source depends
* on driver of your Video4Linux device so this
* may have to be changed */
#define VIDEO_SRC "v4lsrc"
#define VIDEO_SINK "ximagesink"
#endif
/* Define structure for variables that
* are needed thruout the application */
typedef struct
{
HildonProgram *program;
HildonWindow *window;
GstElement *pipeline;
GtkWidget *screen;
guint buffer_cb_id;
} AppData;
static gboolean create_jpeg(unsigned char *buffer);
Then follows the initialize_pipeline() function. This is the heart of our example since it is responsible by creating the GStreamer pipeline, sourcing data from Video4Linux and sinking it to a xvimagesink (which is a optimized X framebuffer). The pipeline scheme is as follows:
|Screen| |Screen|
->|queue |->|sink |-> Display
|Camera| |CSP | |Tee|/
|src |->|Filter|->| |\ |Image | |Image | |Image|
->|queue |->|filter|->|sink |-> JPEG file
Between the source and sinks, we have two ffmpegcolorspace filters, one to configure the camera framerate and the picture size expected by the JPEG encoder, and the second to satisfy the video sink. We employ capabilities ("caps") to tell which format the data needs to have when it exits the filter.
The second filter is necessary since the video sink may have different requirements (bit depth, colorspace) than the JPEG encoder. Also, those requirements are variable accordingly to the hardware.
Since we have two sinks, the queues are important since they guarantee that each pipeline segment downstream the queue operates on its own thread, which is important for the different sinks to synchronize without waiting for each other.
Always remember that this sample application is not different than other GStreamer applications, be it Linux-generic or maemo-specific apps:
static gboolean initialize_pipeline(AppData *appdata,
int *argc, char ***argv)
{
GstElement *pipeline, *camera_src, *screen_sink, *image_sink;
GstElement *screen_queue, *image_queue;
GstElement *csp_filter, *image_filter, *tee;
GstCaps *caps;
GstBus *bus;
/* Initialize Gstreamer */
gst_init(argc, argv);
/* Create pipeline and attach a callback to it's
* message bus */
pipeline = gst_pipeline_new("test-camera");
bus = gst_pipeline_get_bus(GST_PIPELINE(pipeline));
gst_bus_add_watch(bus, (GstBusFunc)bus_callback, appdata);
gst_object_unref(GST_OBJECT(bus));
/* Save pipeline to the AppData structure */
appdata->pipeline = pipeline;
/* Create elements */
/* Camera video stream comes from a Video4Linux driver */
camera_src = gst_element_factory_make(VIDEO_SRC, "camera_src");
/* Colorspace filter is needed to make sure that sinks understands
* the stream coming from the camera */
csp_filter = gst_element_factory_make("ffmpegcolorspace", "csp_filter");
/* Tee that copies the stream to multiple outputs */
tee = gst_element_factory_make("tee", "tee");
/* Queue creates new thread for the stream */
screen_queue = gst_element_factory_make("queue", "screen_queue");
/* Sink that shows the image on screen. Xephyr doesn't support XVideo
* extension, so it needs to use ximagesink, but the device uses
* xvimagesink */
screen_sink = gst_element_factory_make(VIDEO_SINK, "screen_sink");
/* Creates separate thread for the stream from which the image
* is captured */
image_queue = gst_element_factory_make("queue", "image_queue");
/* Filter to convert stream to use format that the gdkpixbuf library
* can use */
image_filter = gst_element_factory_make("ffmpegcolorspace", "image_filter");
/* A dummy sink for the image stream. Goes to bitheaven */
image_sink = gst_element_factory_make("fakesink", "image_sink");
/* Check that elements are correctly initialized */
if(!(pipeline && camera_src && screen_sink && csp_filter && screen_queue
&& image_queue && image_filter && image_sink))
{
g_critical("Couldn't create pipeline elements");
return FALSE;
}
/* Set image sink to emit handoff-signal before throwing away
* it's buffer */
g_object_set(G_OBJECT(image_sink),
"signal-handoffs", TRUE, NULL);
/* Add elements to the pipeline. This has to be done prior to
* linking them */
gst_bin_add_many(GST_BIN(pipeline), camera_src, csp_filter,
tee, screen_queue, screen_sink, image_queue,
image_filter, image_sink, NULL);
/* Specify what kind of video is wanted from the camera */
caps = gst_caps_new_simple("video/x-raw-rgb",
"width", G_TYPE_INT, 640,
"height", G_TYPE_INT, 480,
NULL);
/* Link the camera source and colorspace filter using capabilities
* specified */
if(!gst_element_link_filtered(camera_src, csp_filter, caps))
{
return FALSE;
}
gst_caps_unref(caps);
/* Connect Colorspace Filter -> Tee -> Screen Queue -> Screen Sink
* This finalizes the initialization of the screen-part of the pipeline */
if(!gst_element_link_many(csp_filter, tee, screen_queue, screen_sink, NULL))
{
return FALSE;
}
/* gdkpixbuf requires 8 bits per sample which is 24 bits per
* pixel */
caps = gst_caps_new_simple("video/x-raw-rgb",
"width", G_TYPE_INT, 640,
"height", G_TYPE_INT, 480,
"bpp", G_TYPE_INT, 24,
"depth", G_TYPE_INT, 24,
NULL);
/* Link the image-branch of the pipeline. The pipeline is
* ready after this */
if(!gst_element_link_many(tee, image_queue, image_filter, NULL)) return FALSE;
if(!gst_element_link_filtered(image_filter, image_sink, caps)) return FALSE;
gst_caps_unref(caps);
/* As soon as screen is exposed, window ID will be advised to the sink */
g_signal_connect(appdata->screen, "expose-event", G_CALLBACK(expose_cb),
screen_sink);
gst_element_set_state(pipeline, GST_STATE_PLAYING);
return TRUE;
}
The pipeline also needs a destructor that gets called when the program terminates:
/* Destroy the pipeline on exit */
static void destroy_pipeline(GtkWidget *widget, AppData *appdata)
{
/* Free the pipeline. This automatically also unrefs all elements
* added to the pipeline */
gst_element_set_state(appdata->pipeline, GST_STATE_NULL);
gst_object_unref(GST_OBJECT(appdata->pipeline));
}
The buffer_probe_callback function is called back when the fakesink has data. It will be connected in take_photo-function to the fakesink when user presses the "Take photo" button. The signal connection is removed after the operation is done so that only one photo is taken. The photo itself is created when buffer_probe_callback forwards the buffer to create_jpeg-function, which creates a jpeg-file of the data. It also sends a message whether the operation was succeeded or not to the pipeline's message bus. User can't be informed here because this callback gets called in a gstreamer thread and calling GUI-functions here would lead to problems. :
/* This callback will be registered to the image sink
* after user requests a photo */
static gboolean buffer_probe_callback(
GstElement *image_sink,
GstBuffer *buffer, GstPad *pad, AppData *appdata)
{
GstMessage *message;
gchar *message_name;
/* This is the raw RGB-data that image sink is about
* to discard */
unsigned char *data_photo =
(unsigned char *) GST_BUFFER_DATA(buffer);
/* Create a JPEG of the data and check the status */
if(!create_jpeg(data_photo))
message_name = "photo-failed";
else
message_name = "photo-taken";
/* Disconnect the handler so no more photos
* are taken */
g_signal_handler_disconnect(G_OBJECT(image_sink),
appdata->buffer_cb_id);
/* Create and send an application message which will be
* catched in the bus watcher function. This has to be
* sent as a message because this callback is called in
* a gstreamer thread and calling GUI-functions here would
* lead to X-server synchronization problems */
message = gst_message_new_application(GST_OBJECT(appdata->pipeline),
gst_structure_new(message_name, NULL));
gst_element_post_message(appdata->pipeline, message);
/* Returning TRUE means that the buffer can is OK to be
* sent forward. When using fakesink this doesn't really
* matter because the data is discarded anyway */
return TRUE;
}
/* Callback that gets called when user clicks the "Take photo" button */
static void take_photo(GtkWidget *widget, AppData *appdata)
{
GstElement *image_sink;
/* Get the image sink element from the pipeline */
image_sink = gst_bin_get_by_name(GST_BIN(appdata->pipeline),
"image_sink");
/* Display a note to the user */
hildon_banner_show_information(GTK_WIDGET(appdata->window),
NULL, "Taking Photo");
/* Connect the "handoff"-signal of the image sink to the
* callback. This gets called whenever the sink gets a
* buffer it's ready to pass forward on the pipeline */
appdata->buffer_cb_id = g_signal_connect(
G_OBJECT(image_sink), "handoff",
G_CALLBACK(buffer_probe_callback), appdata);
}
The bus callback gets called whenever the pipeline sends a message thru it's message bus. This handler reports errors and warnings to the console and also reports the user about the status of the photo operation. The reporting has to be done in the main thread and the bus callback gets called in this proper thread. This serves as an example of how a Gstreamer-application should communicate between the different threads.
/* Callback that gets called whenever pipeline's message bus has
* a message */
static void bus_callback(GstBus *bus, GstMessage *message, AppData *appdata)
{
gchar *message_str;
const gchar *message_name;
GError *error;
/* Report errors to the console */
if(GST_MESSAGE_TYPE(message) == GST_MESSAGE_ERROR)
{
gst_message_parse_error(message, &error, &message_str);
g_error("GST error: %s\n", message_str);
g_free(error);
g_free(message_str);
}
/* Report warnings to the console */
if(GST_MESSAGE_TYPE(message) == GST_MESSAGE_WARNING)
{
gst_message_parse_warning(message, &error, &message_str);
g_warning("GST warning: %s\n", message_str);
g_free(error);
g_free(message_str);
}
/* See if the message type is GST_MESSAGE_APPLICATION which means
* thet the message is sent by the client code (this program) and
* not by gstreamer. */
if(GST_MESSAGE_TYPE(message) == GST_MESSAGE_APPLICATION)
{
/* Get name of the message's structure */
message_name = gst_structure_get_name(gst_message_get_structure(message));
/* The hildon banner must be shown in here, because the bus callback is
* called in the main thread and calling GUI-functions in gstreamer threads
* usually leads to problems with X-server */
/* "photo-taken" message means that the photo was succefully taken
* and saved and message is shown to user */
if(!strcmp(message_name, "photo-taken"))
{
hildon_banner_show_information(
GTK_WIDGET(appdata->window),
NULL, "Photo taken");
}
/* "photo-failed" means that the photo couldn't be captured or saved */
if(!strcmp(message_name, "photo-failed"))
{
hildon_banner_show_information(
GTK_WIDGET(appdata->window),
"gtk-dialog-error",
"Saving photo failed");
}
}
}
The xvimagesink GStreamer module will normally create a new window just for itself. Since we want it to show the video inside the main application's window, we need to pass the X-Window window ID to the module, as soon as the ID exists:
/* Callback to be called when the screen-widget is exposed */
static gboolean expose_cb(GtkWidget * widget, GdkEventExpose * event, gpointer data)
{
/* Tell the xvimagesink/ximagesink the x-window-id of the screen
* widget in which the video is shown. After this the video
* is shown in the correct widget */
gst_x_overlay_set_xwindow_id(GST_X_OVERLAY(data),
GDK_WINDOW_XWINDOW(widget->window));
return FALSE;
}
For the sake of completeness, it follows the JPEG encoding function. It is worthwhile to mention that the buffer that came from GStreamer is a simple linear framebuffer:
/* Creates a jpeg file from the buffer's raw image data */
static gboolean create_jpeg(unsigned char *data)
{
GdkPixbuf *pixbuf = NULL;
GError *error = NULL;
guint height, width, bpp;
const gchar *directory;
GString *filename;
guint base_len, i;
struct stat statbuf;
width = 640; height = 480; bpp = 24;
/* Define the save folder */
directory = SAVE_FOLDER_DEFAULT;
if(directory == NULL)
{
directory = g_get_tmp_dir();
}
/* Create an unique file name */
filename = g_string_new(g_build_filename(directory, PHOTO_NAME_DEFAULT, NULL));
base_len = filename->len;
g_string_append(filename, PHOTO_NAME_SUFFIX_DEFAULT);
for(i = 1; !stat(filename->str, &statbuf); ++i)
{
g_string_truncate(filename, base_len);
g_string_append_printf(filename, "%d%s", i, PHOTO_NAME_SUFFIX_DEFAULT);
}
/* Create a pixbuf object from the data */
pixbuf = gdk_pixbuf_new_from_data(data,
GDK_COLORSPACE_RGB, /* RGB-colorspace */
FALSE, /* No alpha-channel */
bpp/3, /* Bits per RGB-component */
width, height, /* Dimensions */
3*width, /* Number of bytes between lines (ie stride) */
NULL, NULL); /* Callbacks */
/* Save the pixbuf content's in to a jpeg file and check for
* errors */
if(!gdk_pixbuf_save(pixbuf, filename->str, "jpeg", &error, NULL))
{
g_warning("%s\n", error->message);
g_error_free(error);
gdk_pixbuf_unref(pixbuf);
g_string_free(filename, TRUE);
return FALSE;
}
/* Free allocated resources and return TRUE which means
* that the operation was succesful */
g_string_free(filename, TRUE);
gdk_pixbuf_unref(pixbuf);
return TRUE;
}
And finally a main()-function which calls the initialization function above, connects some signals, creates the GUI and starts the main loop.
int main(int argc, char **argv)
{
AppData appdata;
GtkWidget *button, *hbox, *vbox_button, *vbox;
/* Initialize and create the GUI */
example_gui_initialize(
&appdata.program, &appdata.window,
&argc, &argv, "Camera example");
vbox = gtk_vbox_new(FALSE, 0);
hbox = gtk_hbox_new(FALSE, 0);
vbox_button = gtk_vbox_new(FALSE, 0);
gtk_box_pack_start(GTK_BOX(hbox), vbox, FALSE, FALSE, 0);
gtk_box_pack_start(GTK_BOX(hbox), vbox_button, FALSE, FALSE, 0);
appdata.screen = gtk_drawing_area_new();
gtk_widget_set_size_request(appdata.screen, 500, 380);
gtk_box_pack_start(GTK_BOX(vbox), appdata.screen, FALSE, FALSE, 0);
button = gtk_button_new_with_label("Take photo");
gtk_widget_set_size_request(button, 170, 380);
gtk_box_pack_start(GTK_BOX(vbox_button), button, FALSE, FALSE, 0);
g_signal_connect(G_OBJECT(button), "clicked",
G_CALLBACK(take_photo), &appdata);
gtk_container_add(GTK_CONTAINER(appdata.window), hbox);
/* Initialize the GTK pipeline */
if(!initialize_pipeline(&appdata, &argc, &argv))
{
hildon_banner_show_information(
GTK_WIDGET(appdata.window),
"gtk-dialog-error",
"Failed to initialize pipeline");
}
g_signal_connect(G_OBJECT(appdata.window), "destroy",
G_CALLBACK(destroy_pipeline), &appdata);
/* Begin the main application */
example_gui_run(appdata.program, appdata.window);
/* Free the gstreamer resources. Elements added
* to the pipeline will be freed automatically */
return 0;
}
Camera manipulation in the Python language
Now, we have a Python example of camera manipulation. It is a simple "digital mirror" that shows the video data in the screen. This example can not take photos since maemo does not have jpeglib bindings for Python yet.
You can also download the complete Python sample.
The main program code is similar to the main() from the previous example. Apart from the Python syntax, the GStreamer pipeline is exactly the same as the one that exists in the C example.
Thanks to Python flexibility, this example will run both in maemo or in a desktop PC without modifications. If you don't have a Video4Linux 2 compatible webcam, you can use videotestsrc as a surrogate video source.
The gst.caps_from_string() will pass the string directly to GStreamer, so don't expect Python layer to check it for you. Note that framebuffer parameter is passed as a fraction, that is, one integer number dividing another one. If you pass a framerate/ resolution combination that the camera cannot handle, no image is shown at all in the screen.
import gtk
try:
import hildon
except ImportError:
hildon = None
import pygst
pygst.require("0.10")
import gst
import time
if hildon:
window = hildon.Window()
else:
window = gtk.Window()
window.set_title("Python Mirror")
window.connect("delete_event", delete_event)
window.connect("destroy", destroy)
window.set_border_width(5)
screen = gtk.DrawingArea()
screen.set_size_request(500, 380)
screen.add_events(gtk.gdk.BUTTON_PRESS_MASK)
screen.connect("expose-event", expose_cb, sink);
screen.connect("button_press_event", click_cb);
window.add(screen)
pipeline = gst.Pipeline("mypipeline")
src = gst.element_factory_make("v4l2src", "src")
pipeline.add(src)
csp = gst.element_factory_make("ffmpegcolorspace", "csp")
pipeline.add(csp)
csp2 = gst.element_factory_make("ffmpegcolorspace", "csp2")
pipeline.add(csp2)
tee = gst.element_factory_make("tee", "tee")
pipeline.add(tee)
queue1 = gst.element_factory_make("queue", "queue1")
pipeline.add(queue1)
queue2 = gst.element_factory_make("queue", "queue2")
pipeline.add(queue2)
videosink = gst.element_factory_make("xvimagesink", "videosink")
pipeline.add(videosink)
fakesink = gst.element_factory_make("fakesink", "fakesink")
pipeline.add(fakesink)
src.link(csp)
csp.link(tee, gst.caps_from_string("video/x-raw-rgb,width=640,height=480,bpp=24,depth=24,framerate=15/1"))
tee.link(queue1)
queue1.link(csp2)
csp2.link(videosink, gst.caps_from_string("video/x-raw-yuv"))
tee.link(queue2)
queue2.link(fakesink)
window.show_all()
fakesink.set_property("signal-handoffs", True)
fakesink.connect("handoff", buffer_cb)
pipeline.set_state(gst.STATE_PLAYING)
gtk.main()
The next functions are boilerplate callbacks. Without them, the Python application would not quit when the Gtk/Hildon window is closed.
The destroy() callback also is in charge of stopping the GStreamer pipeline. It is important to stop it before the window is destroyed, otherwise the video sink would show an error message. (The C version has this specific bug.)
def delete_event(widget, event, data=None): return False def destroy(widget, data=None): pipeline.set_state(gst.STATE_NULL) gtk.main_quit()
As it happens with the C counterpart, we need to inform GStreamer about the drawing area's X-Window ID, so the xvimagesink will show the image on our window instead of creating a new one.
def expose_cb(dummy1, dummy2, dummy3): print "Expose event" sink.set_xwindow_id(screen.window.xid)
As said before, our Python sample is not capable of taking pictures, but we have left the placeholder for such improvement. buffer_cb() is analog to the C version's cb_have_data() callback.
def buffer_cb(element, buffer, pad): # Here you have a chance to do something with the image buffer # By the way, the buffer is received as a Python string pass
Since we don't have any buttons in the Python sample, we need a way to detect user input for e.g. a future picture taking improvement. We have chosen to detect a mouse click in the image screen itself. This is unique to the Python version (the C version has a button for that).
(GTK's DrawingArea does not listen to mouse clicks by default, so this event had to be enabled explicitely at window creation.)
def click_cb(dummy1, dummy2): print "Clicked" return True
Code examples
The examples used in this howto are also part of the maemo-examples package and can be downloaded from the repository.
Improve this page