Planet maemo

madman2k

I recently migrated a server to a new VHost that was supposed to improve the performance – however after the upgrade the performance actually was worse.

Looking at the system load I discovered that the load average was at about 3.5 – with only 2 cores available this corresponds to server overload by almost 2x.

Further looking at the logs revealed that this unfortunately was not due to the users taking interest in the site, but due to various bots hammering on the server. Actual users would be probably drawn away by the awful page load times at this point.

Asking the bots to leave

To improve page loading times, I configured my robots.txt as following

User-agent: *
Disallow: /

This effectively tells all bots to skip my site. You should not do this as you will not be discoverable at e.g. Google.

But here I just wanted to allow my existing users to use the site. Unfortunately the situation only slightly improve; the system load was still over 2.

From the logs I could tell that all bots were actually gone, except for

  • SemrushBot by semrush.com
  • MJ12Bot by majestic.com
  • DotBot by Moz.com

But those were enough to keep the site (PHP+MySQL) overloaded.

The above bots crawl the web for their respective SEO analytics company which sell this information to webmasters. This means that unless you are already a customer of these companies, you do not benefit from having your site crawled.

In fact, if you are interested in SEO analytics for your website, you should probably look elsewhere. In the next paragraph we will block these bots and I am by far not the first one recommending this.

Making the bots leave

As the bots do not respect the robots.txt, you will have to forcefully block them. Instead of the actual webpages, we will give them a 410/ 403 which prevents them touching any PHP/ MySQL resources.

On nginx, add this to your server section:

if ($http_user_agent ~* (SemrushBot|MJ12Bot|DotBot)) {
     return 410;
}

For Apache2.4+ do:

BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase MJ12Bot bad_bot
BrowserMatchNoCase DotBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

For additional fun you could also given them a 307 (redirect) to their own websites here.

Categories: News
mosen

Q1 2018 Community Council Election Announcement

2017-11-07 21:04 UTC  by  mosen
0
0
Dear Maemoans and fellow humans. The time has come again to elect a new Community Council for Q1/2018. The schedule for the voting process is as follows, according to the election rules: The nomination period of at least 2 weeks starts tomorrow, on the 8th of November 2017 and will continue until the 3rd of December 2017. The one week election starts on Friday, the 8th of December 2017 and will continue until the 14th of December 2017. In order for us to keep the community strong, it would be great to have new people with fresh ideas to carry on the torch. Please consider volunteering for the position of Maemo Council. On behalf of the outgoing Community Council, mosen
madman2k

C++ matrix maths – library performance

2017-11-05 01:47 UTC  by  madman2k
0
0

Recently I have been look on the Ogre Matrix class which has a fairly un-optimized, but straightforward implementation, that you can see here.
I was wondering how it compares.

Of course somebody had a similar question in mind before. Martin Foot that is. While the discussion still applies today, I felt like the results could have changed since 2012 as libraries and compilers have moved on.

So I forked his code to update the libs to the latest versions and came up with the following results:

Library add (x86_64, SSSE3) mult (x86_64, SSSE3) add (armeabi-v7a, NEON) mult (armeabi-v7a, NEON) Eigen3 17 ms 53 ms 173 ms 399 ms GLM 50 ms 186 ms 232 ms 399 ms Ogre 50 ms 184 ms 232 ms 399 ms CML1 116 ms 348 ms 178 ms 489 ms

The used compiler was gcc with optimization level -O2.

As we can see Eigen3 just downgrades the rest on x86_64 – probably due its explicit vectorization. Notably, CLM1 is having some issues and even falls behind the naive implementations.
On ARM the results are more tight. With Eigen3 and CLM1 being about 25% faster at addition. However CML1 again has some issues with the mult test.

We end up with Eigen3 being the overall winner and GLM being second (Ogre does not count as it is not a Math library).

Also you should migrate away from CLM1 as the development focus shifted to CLM2 and the issues found above are probably not going to be resolved.

Categories: News
madman2k

Switching Apache2 to php-fpm for performance

2017-10-27 14:43 UTC  by  madman2k
0
0

there are many articles on the internet telling you to switch from Apache & mod_php to nginx to get better performance.

However the main reason for performance improvement is not nginx itself but rather the way it integrates PHP.

Different ways to integrate PHP

Apache traditionally used mod_php to embed the PHP interpreter inside Apache HTTP request handler. This way it can directly interpret PHP scripts whereas with CGI it would have to start a new PHP interpreter process first – per request.

The drawback however is that the PHP interpreter is embedded in all request handlers – even those that just serve static files. This obviously blows up memory consumption which in turn can lower performance.

Nginx on the other hand uses the FCGI approach where a pool of PHP processes is started along the webserver using the FCGI process manager, FPM. The webserver then delegates individual requests using the FCGI protocol as needed.
This avoids the PHP interpreter startup costs as well as starting it without a need and is the reason nginx is faster then mod_php.

However since Apache 2.4 one can also use FCGI to integrate PHP and get virtually the same characteristics like nginx. Sticking with Apache saves you migrating all the .htaccess rules and means an easier setup for many webapps.

Furthermore since Apache 2.4.10 one can use mod_proxy_fcgi for a reverse-proxy configuration which further reduces the occupied PHP workers in the FPM pool for better performance.

Configuration on Ubuntu 16.04

Switching to FCGI on Ubuntu 16.04 is quite easy. The needed module are installed by default and just need to be enabled:

a2enmod proxy_fcgi && a2dismod php7.0

Then inside your-site.conf add

 # PHP-FPM
 <FilesMatch "\.php$">
     SetHandler "proxy:unix:/var/run/php/php7.0-fpm.sock|fcgi://localhost/"
 </FilesMatch>
 <Proxy "fcgi://localhost/">
 </Proxy>

this connects Apache in reverse proxy mode to the PHP-FPM pool using unix domain sockets for optimal performance. See the Apache Wiki for details.

Note that php-fpm by default only creates 5 PHP worker processes, which in turn limits the maximal simultaneous connections. You might want to raise this by adapting pm.max_children in /etc/php/7.0/fpm/pool.d/www.conf.

Typically you set this to RAM size / avg. process size. You can find out the latter via:

ps -ylC php-fpm7.0 --sort:rss
Performance Measurements

To measure the results I did a force reload of my single user Nextcloud instance and measured the Load time via Chrome developer tools:

Page mod_php mod_proxy_fcgi Files 701 ms 605 ms 0.86 News 1.77 s 1.67 s 0.94

as one can see depending on the amount of static/ dynamic files and internal/ external requests we can bring down the page load time by up to 15%.

Categories: News
Philip Van Hoof

Asynchronous commands

2017-10-23 19:31 UTC  by  Philip Van Hoof
0
0

With asynchronous commands we have typical commands from the Model View ViewModel world that return asynchronously.

Whenever that happens we want result reporting and progress reporting. We basically want something like this in QML:

Item {
  id: container
  property ViewModel viewModel: ViewModel {}

  Connections {
    target: viewModel.asyncHelloCommand
    onExecuteProgressed: {
        progressBar.value = value
        progressBar.maximumValue = maximum
    }
  }
  ProgressBar {
     id: progressBar
  }
  Button {
    enabled: viewModel.asyncHelloCommand.canExecute
    onClicked: viewModel.asyncHelloCommand.execute()
  }
}

How do we do this? First we start with defining a AbstractAsyncCommand (impl. of protected APIs here):

class AbstractAsyncCommand : public AbstractCommand {
    Q_OBJECT
public:
    AbstractAsyncCommand(QObject *parent=0);

    Q_INVOKABLE virtual QFuture<void*> executeAsync() = 0;
    virtual void execute() Q_DECL_OVERRIDE;
signals:
    void executeFinished(void* result);
    void executeProgressed(int value, int maximum);
protected:
    QSharedPointer<QFutureInterface<void*>> start();
    void progress(QSharedPointer<QFutureInterface<void*>> fut, int value, int total);
    void finish(QSharedPointer<QFutureInterface<void*>> fut, void* result);
private:
    QVector<QSharedPointer<QFutureInterface<void*>>> m_futures;
};

After that we provide an implementation:

#include <QThreadPool>
#include <QRunnable>

#include <MVVM/Commands/AbstractAsyncCommand.h>

class AsyncHelloCommand: public AbstractAsyncCommand
{
    Q_OBJECT
public:
    AsyncHelloCommand(QObject *parent=0);
    bool canExecute() const Q_DECL_OVERRIDE { return true; }
    QFuture<void*> executeAsync() Q_DECL_OVERRIDE;
private:
    void* executeAsyncTaskFunc();
    QSharedPointer<QFutureInterface<void*>> current;
    QMutex mutex;
};

#include "asynchellocommand.h"

#include <QtConcurrent/QtConcurrent>

AsyncHelloCommand::AsyncHelloCommand(QObject* parent)
    : AbstractAsyncCommand(parent) { }

void* AsyncHelloCommand::executeAsyncTaskFunc()
{
    for (int i=0; i<10; i++) {
        QThread::sleep(1);
        qDebug() << "Hello Async!";
        mutex.lock();
        progress(current, i, 10);
        mutex.unlock();
    }
    return nullptr;
}

QFuture<void*> AsyncHelloCommand::executeAsync()
{
    mutex.lock();
    current = start();
    QFutureWatcher<void*>* watcher = new QFutureWatcher<void*>(this);
    connect(watcher, &QFutureWatcher<void*>::progressValueChanged, this, [=]{
        mutex.lock();
        progress(current, watcher->progressValue(), watcher->progressMaximum());
        mutex.unlock();
    });
    connect(watcher, &QFutureWatcher<void*>::finished, this, [=]{
        void* result=watcher->result();
        mutex.lock();
        finish(current, result);
        mutex.unlock();
        watcher->deleteLater();
    });
    watcher->setFuture(QtConcurrent::run(this, &AsyncHelloCommand::executeAsyncTaskFunc));
    QFuture<void*> future = current->future();
    mutex.unlock();

    return future;
}

You can find the complete working example here.

Categories: controversial
Enrique Ocaña González

This weekend I’ll be in Node5 (Prague) presenting our Media Source Extensions platform implementation work in WebKit using GStreamer.

The Media Source Extensions HTML5 specification allows JavaScript to generate media streams for playback and lets the web page have more control on complex use cases such as adaptive streaming.

My plan for the talk is to start with a brief introduction about the motivation and basic usage of MSE. Next I’ll show a design overview of the WebKit implementation of the spec. Then we’ll go through the iterative evolution of the GStreamer platform-specific parts, as well as its implementation quirks and challenges faced during the development. The talk continues with a demo, some clues about the future work and a final round of questions.

Our recent MSE work has been on desktop WebKitGTK+ (the WebKit version powering the Epiphany, aka: GNOME Web), but we also have MSE working on WPE and optimized for a Raspberry Pi 2. We will be showing it in the Igalia booth, in case you want to see it working live.

I’ll be also attending the GStreamer Hackfest the days before. There I plan to work on webm support in MSE, focusing on any issue in the Matroska demuxer or the vp9/opus/vorbis decoders breaking our use cases.

See you there!

UPDATE 2017-10-22:

The talk slides are available at https://eocanha.org/talks/gstconf2017/gstconf-2017-mse.pdf and the video is available at https://gstconf.ubicast.tv/videos/media-source-extension-on-webkit (the rest of the talks here).

Categories: Gnome
Philip Van Hoof

The RelayCommand in Qt

2017-08-24 18:57 UTC  by  Philip Van Hoof
0
0

A few days ago I explained how we can do MVVM techniques like ICommand in Qt.

Click to read 1112 more words
Categories: controversial
Philip Van Hoof

In the .NET XAML world, you have the ICommand, the CompositeCommand and the DelegateCommand. You use these commands to in a declarative way bind them as properties to XAML components like menu items and buttons. You can find an excellent book on this titled Prism 5.0 for WPF.

Click to read 1778 more words
Categories: controversial
madman2k

Do not use Meson

2017-07-25 14:05 UTC  by  madman2k
0
0

Recently the Meson Build System gained some momentum. It is time to stop that.
Not that Meson is a bad piece of software – on the contrary, it is quite well designed.
Still it makes building C/C++ applications worse, by (quoting xkcd) basically creating  this:

It sets out to create a cross-platform, more readable and faster alternative to autotools. But there is already CMake that solves this.

You might say that CMake is ugly, but note that the CMake 2.x you might have tried is not the same CMake 3.x that is available today. Many patterns have improved and are now both more logical and more readable.

Nowadays the difference between Meson and CMake is just a matter of syntactic preference. The Meson authors seem to agree here.

The actual criterion for selecting a build system however should be tooling support and community spread. CMake easily wins here:

After the introduction of the server mode it got native support by QtCreator, CLion, Android Studio (NDK) and even Microsofts Visual Studio. Native means that you do not have to generate any intermediate project files, but the CMakeLists.txt is used directly by the IDE.

On the community spread side we got e.g. KDE, OpenCV, zlib, libpng, freetype and as of recently Boost. These projects using CMake not only guarantees that you can easily use them, but that you can also include them in your build via add_subdirectory such that the become part of your project. This is especially useful if you are cross-compiling – for instance to a Raspberry Pi.

On the other hand, reinventing a wheel that is tailored to the needs of a specific community (Gnome), means that it will fall behind and eventually die. This is what is currently happening to the Vala language that had a similar birth to Meson.

The meson devs might object that Meson generates build files that run faster on a Raspberry Pi. However if your cross compiling is working you do not need that. And honestly, that particular improvement could have been also achieved by providing a patch to the CMake Ninja generator..

 

 

Categories: Articles
Jussi Ohenoja

Meet the new Q2 2017 Maemo Community Council

2017-07-11 20:17 UTC  by  Jussi Ohenoja
0
0

Dear Maemo community, I have the great honor of introducing the new Community Council for the upcoming Q2/2017 period.

**The members of the new council are (in alphabetical order):**

  • Juiceme (Jussi Ohenoja)
  • Mosen (Timo Könnecke)
  • Sicelo (Sicelo Mhlongo)

The voting results can be seen on the [voting page]

I want to thank warmly all the members of the community who participated in this most important action of choosing a new council for us!

The new council shall meet on the #maemo-meeting IRC channel next tuesday 18.06 at 20:00 UTC for the formal handover with the passing council.

Jussi Ohenoja, On behalf of the outgoing Maemo Community Council

Categories: council
Philip Van Hoof

I’m at home now. I don’t do non-public unpaid work. So let’s blog the example I’m making for him.

Click to read 1082 more words
Categories: condescending
Philip Van Hoof

Imagine we want an editor that has undo and redo capability. But the operations on the editor are all asynchronous. This implies that also undo and redo are asynchronous operations.

Click to read 2050 more words
Categories: controversial