When I’m not in Flowhub-land, I’m used to developing software in a quite customized command line based development environment. Like for many, the cornerstones of this for me are vim and tmux.

I recently migrated a server to a new VHost that was supposed to improve the performance – however after the upgrade the performance actually was worse.

Looking at the system load I discovered that the load average was at about 3.5 – with only 2 cores available this corresponds to server overload by almost 2x.

Further looking at the logs revealed that this unfortunately was not due to the users taking interest in the site, but due to various bots hammering on the server. Actual users would be probably drawn away by the awful page load times at this point.

Asking the bots to leave

To improve page loading times, I configured my robots.txt as following

User-agent: *
Disallow: /

This effectively tells all bots to skip my site. You should not do this as you will not be discoverable at e.g. Google.

But here I just wanted to allow my existing users to use the site. Unfortunately the situation only slightly improve; the system load was still over 2.

From the logs I could tell that all bots were actually gone, except for

• SemrushBot by semrush.com
• MJ12Bot by majestic.com
• DotBot by Moz.com

But those were enough to keep the site (PHP+MySQL) overloaded.

The above bots crawl the web for their respective SEO analytics company which sell this information to webmasters. This means that unless you are already a customer of these companies, you do not benefit from having your site crawled.

In fact, if you are interested in SEO analytics for your website, you should probably look elsewhere. In the next paragraph we will block these bots and I am by far not the first one recommending this.

Making the bots leave

As the bots do not respect the robots.txt, you will have to forcefully block them. Instead of the actual webpages, we will give them a 410/ 403 which prevents them touching any PHP/ MySQL resources.

On nginx, add this to your server section:

if ($http_user_agent ~* (SemrushBot|MJ12Bot|DotBot)) {
     return 410;
}

For Apache2.4+ do:

BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase MJ12Bot bad_bot
BrowserMatchNoCase DotBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

For additional fun you could also given them a 307 (redirect) to their own websites here.

Dear Maemoans and fellow humans. The time has come again to elect a new Community Council for Q1/2018. The schedule for the voting process is as follows, according to the election rules: The nomination period of at least 2 weeks starts tomorrow, on the 8th of November 2017 and will continue until the 3rd of December 2017. The one week election starts on Friday, the 8th of December 2017 and will continue until the 14th of December 2017. In order for us to keep the community strong, it would be great to have new people with fresh ideas to carry on the torch. Please consider volunteering for the position of Maemo Council. On behalf of the outgoing Community Council, mosen

Recently I have been look on the Ogre Matrix class which has a fairly un-optimized, but straightforward implementation, that you can see here.
I was wondering how it compares.

Of course somebody had a similar question in mind before. Martin Foot that is. While the discussion still applies today, I felt like the results could have changed since 2012 as libraries and compilers have moved on.

So I forked his code to update the libs to the latest versions and came up with the following results:

Library add (x86_64, SSSE3) mult (x86_64, SSSE3) add (armeabi-v7a, NEON) mult (armeabi-v7a, NEON) Eigen3 17 ms 53 ms 173 ms 399 ms GLM 50 ms 186 ms 232 ms 399 ms Ogre 50 ms 184 ms 232 ms 399 ms CML1 116 ms 348 ms 178 ms 489 ms

The used compiler was gcc with optimization level -O2.

As we can see Eigen3 just downgrades the rest on x86_64 – probably due its explicit vectorization. Notably, CLM1 is having some issues and even falls behind the naive implementations.
On ARM the results are more tight. With Eigen3 and CLM1 being about 25% faster at addition. However CML1 again has some issues with the mult test.

We end up with Eigen3 being the overall winner and GLM being second (Ogre does not count as it is not a Math library).

Also you should migrate away from CLM1 as the development focus shifted to CLM2 and the issues found above are probably not going to be resolved.

there are many articles on the internet telling you to switch from Apache & mod_php to nginx to get better performance.

However the main reason for performance improvement is not nginx itself but rather the way it integrates PHP.

Different ways to integrate PHP

Apache traditionally used mod_php to embed the PHP interpreter inside Apache HTTP request handler. This way it can directly interpret PHP scripts whereas with CGI it would have to start a new PHP interpreter process first – per request.

The drawback however is that the PHP interpreter is embedded in all request handlers – even those that just serve static files. This obviously blows up memory consumption which in turn can lower performance.

Nginx on the other hand uses the FCGI approach where a pool of PHP processes is started along the webserver using the FCGI process manager, FPM. The webserver then delegates individual requests using the FCGI protocol as needed.
This avoids the PHP interpreter startup costs as well as starting it without a need and is the reason nginx is faster then mod_php.

However since Apache 2.4 one can also use FCGI to integrate PHP and get virtually the same characteristics like nginx. Sticking with Apache saves you migrating all the .htaccess rules and means an easier setup for many webapps.

Furthermore since Apache 2.4.10 one can use mod_proxy_fcgi for a reverse-proxy configuration which further reduces the occupied PHP workers in the FPM pool for better performance.

Configuration on Ubuntu 16.04

Switching to FCGI on Ubuntu 16.04 is quite easy. The needed module are installed by default and just need to be enabled:

a2enmod proxy_fcgi && a2dismod php7.0

Then inside your-site.conf add

 # PHP-FPM
 <FilesMatch "\.php$">
     SetHandler "proxy:unix:/var/run/php/php7.0-fpm.sock|fcgi://localhost/"
 </FilesMatch>
 <Proxy "fcgi://localhost/">
 </Proxy>

this connects Apache in reverse proxy mode to the PHP-FPM pool using unix domain sockets for optimal performance. See the Apache Wiki for details.

Note that php-fpm by default only creates 5 PHP worker processes, which in turn limits the maximal simultaneous connections. You might want to raise this by adapting pm.max_children in /etc/php/7.0/fpm/pool.d/www.conf.

Typically you set this to RAM size / avg. process size. You can find out the latter via:

ps -ylC php-fpm7.0 --sort:rss

Performance Measurements

To measure the results I did a force reload of my single user Nextcloud instance and measured the Load time via Chrome developer tools:

Page mod_php mod_proxy_fcgi Files 701 ms 605 ms 0.86 News 1.77 s 1.67 s 0.94

as one can see depending on the amount of static/ dynamic files and internal/ external requests we can bring down the page load time by up to 15%.

With asynchronous commands we have typical commands from the Model View ViewModel world that return asynchronously.

Whenever that happens we want result reporting and progress reporting. We basically want something like this in QML:

Item {
  id: container
  property ViewModel viewModel: ViewModel {}

  Connections {
    target: viewModel.asyncHelloCommand
    onExecuteProgressed: {
        progressBar.value = value
        progressBar.maximumValue = maximum
    }
  }
  ProgressBar {
     id: progressBar
  }
  Button {
    enabled: viewModel.asyncHelloCommand.canExecute
    onClicked: viewModel.asyncHelloCommand.execute()
  }
}

How do we do this? First we start with defining a AbstractAsyncCommand (impl. of protected APIs here):

class AbstractAsyncCommand : public AbstractCommand {
    Q_OBJECT
public:
    AbstractAsyncCommand(QObject *parent=0);

    Q_INVOKABLE virtual QFuture<void*> executeAsync() = 0;
    virtual void execute() Q_DECL_OVERRIDE;
signals:
    void executeFinished(void* result);
    void executeProgressed(int value, int maximum);
protected:
    QSharedPointer<QFutureInterface<void*>> start();
    void progress(QSharedPointer<QFutureInterface<void*>> fut, int value, int total);
    void finish(QSharedPointer<QFutureInterface<void*>> fut, void* result);
private:
    QVector<QSharedPointer<QFutureInterface<void*>>> m_futures;
};

After that we provide an implementation:

#include <QThreadPool>
#include <QRunnable>

#include <MVVM/Commands/AbstractAsyncCommand.h>

class AsyncHelloCommand: public AbstractAsyncCommand
{
    Q_OBJECT
public:
    AsyncHelloCommand(QObject *parent=0);
    bool canExecute() const Q_DECL_OVERRIDE { return true; }
    QFuture<void*> executeAsync() Q_DECL_OVERRIDE;
private:
    void* executeAsyncTaskFunc();
    QSharedPointer<QFutureInterface<void*>> current;
    QMutex mutex;
};

#include "asynchellocommand.h"

#include <QtConcurrent/QtConcurrent>

AsyncHelloCommand::AsyncHelloCommand(QObject* parent)
    : AbstractAsyncCommand(parent) { }

void* AsyncHelloCommand::executeAsyncTaskFunc()
{
    for (int i=0; i<10; i++) {
        QThread::sleep(1);
        qDebug() << "Hello Async!";
        mutex.lock();
        progress(current, i, 10);
        mutex.unlock();
    }
    return nullptr;
}

QFuture<void*> AsyncHelloCommand::executeAsync()
{
    mutex.lock();
    current = start();
    QFutureWatcher<void*>* watcher = new QFutureWatcher<void*>(this);
    connect(watcher, &QFutureWatcher<void*>::progressValueChanged, this, [=]{
        mutex.lock();
        progress(current, watcher->progressValue(), watcher->progressMaximum());
        mutex.unlock();
    });
    connect(watcher, &QFutureWatcher<void*>::finished, this, [=]{
        void* result=watcher->result();
        mutex.lock();
        finish(current, result);
        mutex.unlock();
        watcher->deleteLater();
    });
    watcher->setFuture(QtConcurrent::run(this, &AsyncHelloCommand::executeAsyncTaskFunc));
    QFuture<void*> future = current->future();
    mutex.unlock();

    return future;
}

You can find the complete working example here.

This weekend I’ll be in Node5 (Prague) presenting our Media Source Extensions platform implementation work in WebKit using GStreamer.

The Media Source Extensions HTML5 specification allows JavaScript to generate media streams for playback and lets the web page have more control on complex use cases such as adaptive streaming.

My plan for the talk is to start with a brief introduction about the motivation and basic usage of MSE. Next I’ll show a design overview of the WebKit implementation of the spec. Then we’ll go through the iterative evolution of the GStreamer platform-specific parts, as well as its implementation quirks and challenges faced during the development. The talk continues with a demo, some clues about the future work and a final round of questions.

Our recent MSE work has been on desktop WebKitGTK+ (the WebKit version powering the Epiphany, aka: GNOME Web), but we also have MSE working on WPE and optimized for a Raspberry Pi 2. We will be showing it in the Igalia booth, in case you want to see it working live.

I’ll be also attending the GStreamer Hackfest the days before. There I plan to work on webm support in MSE, focusing on any issue in the Matroska demuxer or the vp9/opus/vorbis decoders breaking our use cases.

See you there!

UPDATE 2017-10-22:

The talk slides are available at https://eocanha.org/talks/gstconf2017/gstconf-2017-mse.pdf and the video is available at https://gstconf.ubicast.tv/videos/media-source-extension-on-webkit (the rest of the talks here).

A few days ago I explained how we can do MVVM techniques like ICommand in Qt.

In the .NET XAML world, you have the ICommand, the CompositeCommand and the DelegateCommand. You use these commands to in a declarative way bind them as properties to XAML components like menu items and buttons. You can find an excellent book on this titled Prism 5.0 for WPF.