AR-Drone with MeeGo

Main view with live feed from AR-Drone camera and telemetry


Here is a little hcking project that I am releasing just before Christmas, it is something fun to play around, something that really moves and makes also non-hackers exited. It also allows to you use own imagination to enhance this base application to be even more fun. Least to me, Christmas has been always time to build something that is fun,like all kind on flying gadgets. I still like playing with everything that flies, some ones where I can sit myself in pilot seat and someone that I can control remotely.
I decided to combine work and fun and also offer a super-cool possibility to learn MeeGo and Qt Quick with having fun, just go shop and buy AR-Drone 299$ and download code from http://mardrone.garage.maemo.org and have happy hacking.
 
After Nokia announcement focus Qt and Qt Quick and after my previous blog howto make modern mobile applications with qt quick components there has been a lot of need for some real life full show case applications how to do UI with Qml and how to use existing C++ application logic. We have also here in Nokia term “dogfooding”, we should eat own dog food that we make. In his Qt Quick / Qt Components in the case I should say that it's the best dog food I have ever had.

Other goal for this application has been show, how easy is to make MeeGo versions of iPhone applications. Original Ar-Drone FreeFlight application was made for iPhone. Making Qt Quick UI for this application takes just day or two. Current UI still looks like engineer's tool, because it is made by an engineer. To make it really good looking needs touch of UI designer and great graphics but no SDK can solve this problem even it is iPhone, Android or MeeGo. Integrating C++ code to Qml took less than days work.  The more detailed anatomy of this application is coming on second part, next blog entry.

There has been also criticism and rumors that Nokia would be killing Maemo and no-one should any more develop for Maemo. I would like to prove all these rumors to be wrong. The best option today is to develop with maemo5 with Qt and Qt Quick and when a MeeGo handset is released, just deploy your application for it, it is that easy.

You should remember that our Harmattan project started as called Maemo6 but after merging to Intel Moblin it became MeeGo. Maemo and Moblin already had common roots and result of this merge was not incompatible what Maemo6 was. I have compiled and run this same application without changing a single line of code in both N900 with Maemo5, our future MeeGo based device and Ubuntu Lucid in desktop. Your application will run on all platforms unless you intentionally chose some obsoleted API's like GTK+ based Hildon. For someone that does not like EOL of some old API's I recommend reading my previous blog entry, we just needed to renew our UI technology to be competitive and as application developers you should be happy that we can provide now best application development platform where your imagination is only limit in UI creation.

AR-Drone and application

Technologies, that I use in this AR-Drone application are: All QI Is Implemented with Qt Quick QML Language and MeeGo Qt Quick Components with addition of some my own QML and C++ components. I have integrated to the UI seamlessly some my existing Qt GraphicsView code as custom component and existing C-code, AR-Drone video codec used in iPhone and in their GTK+ demo application. All application logic is coded with Qt / C++ . Application uses real time and performance critical things like live video feed from drone camera, real time telemetry data from drone and augmentation of video with realtime HUD display.

What is then the AR-Drone ? AR-Drone is quatrocopter, Vertical Take off and Landing ( VTOL) aircraft, rotorcraft like helicopter. Quadrocopters are not so efficient for cross country flying they are very simple to construct with electric motors and due simplicity they are very crash proof and durable. With implementing artificial stabilization with software and sensors they can be made very easy to fly and handle. Typical RC-helicopter may need dozens of hours, many dozens of crashes, disappointments, lot of glue and hundreds of euros. As opposite, AR Drone, you just press fly button and it starts hover about one meter over ground and then you can start moving it.

Technically AR-Drone is much more advanced than standard RC-aircrafts. It has 32 Bit ARM processor running Linux, it is controlled over WLAN and it has bi-directional data stream transmitting back both telemetry data and live video feed from two cameras on drone. For flight control AR-Drone does have 3 axis MEMS Gyro ( angular speed sensor) and linear acceleration sensor, ultrasound for measuring altitude and bottom facing camera for position holding.

AR Stands for augmented reality and it's manufacturer, Parrot calls it as flying video game. You can implement as example air fight between two drones by detecting others color tags from video and augmenting video image with ammunition trails or misles. I my application I did not yet try to make this kind of game but i have already augmenting layer implementing HUD ( Heads Up Diaplay) , similar than fighter planes have showing flight instruments over pilot's vision.

This is not a sponsored add, i just bough my AR-Drone from Sunnyvale Bookstone shop with $299.

Basic structure of application

The main program is very simple, it just registers my three C++ classes to Qml and then creates QdeclatariveView with my main Qml file, ardrone.qml. For easier application packing, it uses Qt resources to keep qml files, so no extra files is needed to install with application. The ardrone.qml then imports my “drone” classes with import Drone 1.0 . 

Main class is called by DroneControl, it does not have any visible representation to Qml, it's purpose is to implement actual application logic and expose it's interface to QML. DroneControl creates instance of  DroneThread that handles real time control and telemetry connection with drone. DroneControl class also implements interface to QSettings for application settings storage.
DroneControl also implements command protocol and telemetry data parsing. They are implemented from scratch with Qt, i had option to use also AR-Drone SDK VP_LIB library functions but they had so heavy dependencies that i did not like and the command protocol was so simple that i decided that it is easier just do own implementation with Qt/C++

As addition to main Logic, I created class DroneVideo derived from QGraphicsWidget . For Qt C++ view it is just like any other QGraphicsWidget derived class that can be used to Qt QGraphicsView applications. It did not need any modifications to be used as component from QML. Standard QGraphicsWidget properties like position, size etc are already automatically exposed to QML. That means that you can use your QGraphicsWidget or QGraphicsItem derived classes in many cases without any modifications from QML. The DroneVideo implements video streaming functionality via separate VideoThread. It was easiest way to implement this application and re-use codec from AR-Drone SDK. In future more elegant solution would be to implement codec as Gstreamer module. The DroneVideo is not yet optimized at all, in future I should also consider implementing YUV-RGB conversion and scaling with GPU OpenGL-ES2 and shader code. Performance currently in N900 is fair but even this implementation gives relativelly good performance in MeeGo device.

Third C++ class I used, was also re-use from my previous Qt QGraphicsView based application implementing HUD, Heads Up display, artificial horizon with declination and inclination indicators.  It was mostly as it is, but just non-standard properties exposed to QML with Q_PROPERTY declaration. HUD is drawn as overlay over video picture. If yoy would like to add some augmented reality stuff like gunfire, missles etc, just replace HUD drawing code with it. You could also use Qt OpenGL-ES2 if you need more compled Augmented reality.

The actual UI is implemented with Qt Quick and Qt Quick MeeGo Components. The main QML UI file is under 200 lines of code, specifying layout of main screen of application and binding UI element properties to C++ class variables. Main page is mostly custom UI in full screen mode, more like games are but it uses MeeGo Qt Components framework and some items like Button and Text there. Joysticks are implemented as my own QML components. If you need some custom UI component, in  many cases you can do it with 10..50 lines of QML. The settings page is implemented as separate page less than 200 lines. For easy application deployment, all qml files are included to executable code as resources, so no extra directory is needed for them.

Building application

To build application, you need to have Qt 4.7 and Qt Quick components installed.  You can read from  howto make modern mobile applications with qt quick components how to install them for both Ubuntu and Maemo5 Scratchbox development environment. You don't need to use Qt Creator but if you would like to use it, you can read it from qt creator and scratchbox .

My application source code you can get from http://mardrone.garage.maemo.org with git.

git clone https://vcs.maemo.org/git/mardrone

To addition, you need AR-Drone SDK that contains video codec library for AR-Drone from
https://projects.ardrone.org/  

You need to apply patch ARDroneLib.patch from mardrone to compile AR-Drone libraries under scratchbox for Maemo/MeeGo target and to expose couple of previously static functions to application. The AR-Drone SDK were unaware that you could compile for ARM target under Linux that looks a like “native” ARM that does not look as embedded linux.

If you would like to build test version of this application for both desktop and Maemo/MeeGo target
and do it automatically from Qt Creator from same source directory, extract your AR-Drone SDK for both your Ubuntu host and scratchbox under your home directory. They are in real life different directories and so you can have x86 in host and ARM under your scratchbox. To build, execute under your host and scratchbox both

cd ARDrone_SDK_1_5_Version_20101004/ARDroneLib/Soft/Build/
make

To build actual application, you can either just open ardrone1.pro file with your Qt Creator and
press run, it should compile it with your host Qt4.7 . Compiling under scratchbox from Qt creator, please refer previous blog. Or you can just compile it from command line

qmake
make

If you would like to make installable debian package:

fakeroot dpkg-buildpackage -b

And you should have ready made application to run. If you have installed it from debian package, you will have ready icon. If you would like to run it fron command line, In  current pre-alpha N900 Qt Quick components MeeGoTouch style libraries there is bug that you should start your application with

mardrone -graphicsystem native 

That is slower than opengl but with opengl render you got all components as red background.

Using using application with AR-Drone

WARNING this application is at the moment early alpha level, version 0.1. It is intended to be used as technology demo and suitable only hackers that can read, understand and fix the code. There are still unknown bugs and unimplemented features like acceleration sensor, joystick, better UI style etc. Don't try this unless you know what you are doing.

When you have your drone, i recommend to first test flight in place where you have free space around but not in windy weather. Best place is big empty room like garage etc. The drone should auto-hover after takeoff and it if crash resistant, so it should not get easily broken if you collide walls etc but try avoid it.

First step, you should connect your device to AR-Drone Ad-Hoc WiFi network. After powering up your drone., you should see network named ardrone_xxxxxx, connect to that one. It should assign IP 192.168.1.2 to your device and drone will be 192.168.1.1 .

The next step is to start mardrone application. It takes little moment and you should see application main window. If the connection to drone is successful, you should see video feed from drone camera. If you see gray background with X, over it, there is no video feed and no connection to drone.

By default you should also see two lines of green text over joysticks. They are debug status from drone. If there is something wrong in your drone, you should see message there. As example, if you carry it by hand, it may have too high angle and you get message there. You can reset status by pressing emergency twice.

To fly the drone, press “fly” button on bottom of the screen. The drone should take off and hover about one meter over ground. It should not move to any direction without you controlling it with joysticks in left and right side of screen. Try first right joystick, it moves drone up, down and turn around. When you are familiar controlling, try right one, it makes drone to fly forward, backward, left and right. Be careful, with full deflection drone flies fast and may collide something.

 

Happy Hacking