Planet maemo: category "feed:dc2d42ffa90d409ad35691447d64bb45"

madman2k

I recently migrated a server to a new VHost that was supposed to improve the performance – however after the upgrade the performance actually was worse.

Looking at the system load I discovered that the load average was at about 3.5 – with only 2 cores available this corresponds to server overload by almost 2x.

Further looking at the logs revealed that this unfortunately was not due to the users taking interest in the site, but due to various bots hammering on the server. Actual users would be probably drawn away by the awful page load times at this point.

Asking the bots to leave

To improve page loading times, I configured my robots.txt as following

User-agent: *
Disallow: /

This effectively tells all bots to skip my site. You should not do this as you will not be discoverable at e.g. Google.

But here I just wanted to allow my existing users to use the site. Unfortunately the situation only slightly improve; the system load was still over 2.

From the logs I could tell that all bots were actually gone, except for

  • SemrushBot by semrush.com
  • MJ12Bot by majestic.com
  • DotBot by Moz.com

But those were enough to keep the site (PHP+MySQL) overloaded.

The above bots crawl the web for their respective SEO analytics company which sell this information to webmasters. This means that unless you are already a customer of these companies, you do not benefit from having your site crawled.

In fact, if you are interested in SEO analytics for your website, you should probably look elsewhere. In the next paragraph we will block these bots and I am by far not the first one recommending this.

Making the bots leave

As the bots do not respect the robots.txt, you will have to forcefully block them. Instead of the actual webpages, we will give them a 410/ 403 which prevents them touching any PHP/ MySQL resources.

On nginx, add this to your server section:

if ($http_user_agent ~* (SemrushBot|MJ12Bot|DotBot)) {
     return 410;
}

For Apache2.4+ do:

BrowserMatchNoCase SemrushBot bad_bot
BrowserMatchNoCase MJ12Bot bad_bot
BrowserMatchNoCase DotBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

For additional fun you could also given them a 307 (redirect) to their own websites here.

Categories: News
madman2k

C++ matrix maths – library performance

2017-11-05 01:47 UTC  by  madman2k
0
0

Recently I have been look on the Ogre Matrix class which has a fairly un-optimized, but straightforward implementation, that you can see here.
I was wondering how it compares.

Of course somebody had a similar question in mind before. Martin Foot that is. While the discussion still applies today, I felt like the results could have changed since 2012 as libraries and compilers have moved on.

So I forked his code to update the libs to the latest versions and came up with the following results:

Library add (x86_64, SSSE3) mult (x86_64, SSSE3) add (armeabi-v7a, NEON) mult (armeabi-v7a, NEON) Eigen3 17 ms 53 ms 173 ms 399 ms GLM 50 ms 186 ms 232 ms 399 ms Ogre 50 ms 184 ms 232 ms 399 ms CML1 116 ms 348 ms 178 ms 489 ms

The used compiler was gcc with optimization level -O2.

As we can see Eigen3 just downgrades the rest on x86_64 – probably due its explicit vectorization. Notably, CLM1 is having some issues and even falls behind the naive implementations.
On ARM the results are more tight. With Eigen3 and CLM1 being about 25% faster at addition. However CML1 again has some issues with the mult test.

We end up with Eigen3 being the overall winner and GLM being second (Ogre does not count as it is not a Math library).

Also you should migrate away from CLM1 as the development focus shifted to CLM2 and the issues found above are probably not going to be resolved.

Categories: News
madman2k

Switching Apache2 to php-fpm for performance

2017-10-27 14:43 UTC  by  madman2k
0
0

there are many articles on the internet telling you to switch from Apache & mod_php to nginx to get better performance.

However the main reason for performance improvement is not nginx itself but rather the way it integrates PHP.

Different ways to integrate PHP

Apache traditionally used mod_php to embed the PHP interpreter inside Apache HTTP request handler. This way it can directly interpret PHP scripts whereas with CGI it would have to start a new PHP interpreter process first – per request.

The drawback however is that the PHP interpreter is embedded in all request handlers – even those that just serve static files. This obviously blows up memory consumption which in turn can lower performance.

Nginx on the other hand uses the FCGI approach where a pool of PHP processes is started along the webserver using the FCGI process manager, FPM. The webserver then delegates individual requests using the FCGI protocol as needed.
This avoids the PHP interpreter startup costs as well as starting it without a need and is the reason nginx is faster then mod_php.

However since Apache 2.4 one can also use FCGI to integrate PHP and get virtually the same characteristics like nginx. Sticking with Apache saves you migrating all the .htaccess rules and means an easier setup for many webapps.

Furthermore since Apache 2.4.10 one can use mod_proxy_fcgi for a reverse-proxy configuration which further reduces the occupied PHP workers in the FPM pool for better performance.

Configuration on Ubuntu 16.04

Switching to FCGI on Ubuntu 16.04 is quite easy. The needed module are installed by default and just need to be enabled:

a2enmod proxy_fcgi && a2dismod php7.0

Then inside your-site.conf add

 # PHP-FPM
 <FilesMatch "\.php$">
     SetHandler "proxy:unix:/var/run/php/php7.0-fpm.sock|fcgi://localhost/"
 </FilesMatch>
 <Proxy "fcgi://localhost/">
 </Proxy>

this connects Apache in reverse proxy mode to the PHP-FPM pool using unix domain sockets for optimal performance. See the Apache Wiki for details.

Note that php-fpm by default only creates 5 PHP worker processes, which in turn limits the maximal simultaneous connections. You might want to raise this by adapting pm.max_children in /etc/php/7.0/fpm/pool.d/www.conf.

Typically you set this to RAM size / avg. process size. You can find out the latter via:

ps -ylC php-fpm7.0 --sort:rss
Performance Measurements

To measure the results I did a force reload of my single user Nextcloud instance and measured the Load time via Chrome developer tools:

Page mod_php mod_proxy_fcgi Files 701 ms 605 ms 0.86 News 1.77 s 1.67 s 0.94

as one can see depending on the amount of static/ dynamic files and internal/ external requests we can bring down the page load time by up to 15%.

Categories: News
madman2k

Do not use Meson

2017-07-25 14:05 UTC  by  madman2k
0
0

Recently the Meson Build System gained some momentum. It is time to stop that.
Not that Meson is a bad piece of software – on the contrary, it is quite well designed.
Still it makes building C/C++ applications worse, by (quoting xkcd) basically creating  this:

It sets out to create a cross-platform, more readable and faster alternative to autotools. But there is already CMake that solves this.

You might say that CMake is ugly, but note that the CMake 2.x you might have tried is not the same CMake 3.x that is available today. Many patterns have improved and are now both more logical and more readable.

Nowadays the difference between Meson and CMake is just a matter of syntactic preference. The Meson authors seem to agree here.

The actual criterion for selecting a build system however should be tooling support and community spread. CMake easily wins here:

After the introduction of the server mode it got native support by QtCreator, CLion, Android Studio (NDK) and even Microsofts Visual Studio. Native means that you do not have to generate any intermediate project files, but the CMakeLists.txt is used directly by the IDE.

On the community spread side we got e.g. KDE, OpenCV, zlib, libpng, freetype and as of recently Boost. These projects using CMake not only guarantees that you can easily use them, but that you can also include them in your build via add_subdirectory such that the become part of your project. This is especially useful if you are cross-compiling – for instance to a Raspberry Pi.

On the other hand, reinventing a wheel that is tailored to the needs of a specific community (Gnome), means that it will fall behind and eventually die. This is what is currently happening to the Vala language that had a similar birth to Meson.

The meson devs might object that Meson generates build files that run faster on a Raspberry Pi. However if your cross compiling is working you do not need that. And honestly, that particular improvement could have been also achieved by providing a patch to the CMake Ninja generator..

 

 

Categories: Articles
madman2k

Migrating from owncloud 9.1 to nextcloud 11

2017-02-10 23:33 UTC  by  madman2k
0
0

First one should ask though: why? My main motivation was that many of the apps I use were easily available in the nextcloud store, while with owncloud I had to manually pull them from github.
Additionally some of the app authors migrated to nextcloud and did not provide further updates for owncloud.

Another reason is this:

the graphs above show the number of commits for owncloud and nextcloud. Owncloud has taken a very noticeable hit here after the fork – even though they deny it.

From the user perspective the lack of contribution is visible for instance in the admin interface where with nextcloud you get a nice log browser and system stats while with owncloud you do not. Furthermore the nextcloud android app handles Auto-Upload much better and generally seems more polished – I think one can expect nextcloud to advance faster in general.

Migrating

For migrating you can follow the excellent instructions of Jos Poortvliet.

In my case owncloud 9.1 was installed on Ubuntu in /var/www/owncloud and I put nextcloud 11 to /var/www/nextcloud. Then the following steps had to be applied:

  1. put owncloud in maintenance mode
    sudo -u www-data php occ maintenance:mode --on
  2. copy over the config.php
    cp /var/www/owncloud/config/config.php /var/www/nextcloud/config/
  3. adapt the path in config.php
    # from 
    'path' => '/var/www/owncloud/apps',
    # to
    'path' => '/var/www/nextcloud/apps',
  4. adapt the path in crontab
    sudo crontab -u www-data -e
  5. adapt the paths in the apache config
  6. run the upgrade script which takes care of the actual migration. Then disable the maintanance mode.
    sudo -u www-data php occ upgrade
    sudo -u www-data php occ maintenance:mode --off

and thats it.

Categories: News
madman2k

OGRECave 1.10 release

2016-12-22 01:26 UTC  by  madman2k
0
0

The 1.10.0 release of the OGRECave fork was just created. This means that the code is considered stable enough for general usage and the current interfaces will be supported in subsequent patch releases (i.e. 1.10.1, 1.10.2 …).

SampleBrowser running GLES2 on desktop

This release represents more than 3 years of work from various contributors when compared to the previous 1.9 release. At the time of writing it contains all commits from the bitbucket version as well as many fork specific features and fixes.

If you are reading about the fork for the first time and wonder why it was created, see this blog post. For a comparison between the github and bitbucket version see this log.

For a general overview of the 1.10 features when compared to 1.9, see the OGRECave 1.10 release notes.

The highlights probably are:

  • upstream Python bindings as an component
  • improved GL3+/ GLES2 renderers
  • A new HLMS Component implementing physically based shading
  • SDL2 based input handling
  • Bites Component for rapid prototyping of applications
  • Emscripten platform support

For further information see the github page of the fork.

Categories: Graphics
madman2k

On OGRE versions

2016-08-05 12:24 UTC  by  madman2k
0
0

Currently one can choose between the following OGRE versions
1.9, 1.10, 2.0 and 2.1

Click to read 1288 more words
Categories: Graphics
madman2k

Creating PyGTK app snaps with snapcraft

2016-07-09 13:45 UTC  by  madman2k
0
0

Snap is a new packaging format introduced by Ubuntu as an successor to dpkg aka debian package. It offers sandboxing and transactional updates and thus is a competitor to the flatpak format and resembles docker images.

Click to read 1132 more words
Categories: Articles
madman2k

Introducing the OGRE fork on GitHub

2016-05-02 14:38 UTC  by  madman2k
0
0

in this post I want to introduce the OGRE fork on github. The goal of the fork is to provide a stable and reliable OGRE 1.x series while at the same time modernizing parts under the hood updates.

Click to read 1522 more words
Categories: Graphics
madman2k

Learning Modern 3D Graphics Programming

2015-12-29 22:26 UTC  by  madman2k
0
0

one of the best resources to learn modern OpenGL and the one which helped me quite a lot is the Book at www.arcsynthesis.org/gltut/ – or lets better say was. Unfortunately the domain expired so the content is no longer reachable.

Luckily the Book was designed as an open source project and the code to generate the website is still available at Bitbucket. Unfortunately this repository does not seem to be actively maintained any more.

Therefore I set out to make the Book to be available again using Github Pages. You can find the results here:

https://paroj.github.io/gltut/

However I did not simply mirror the pages, but also improved it at several places. So what has changed so far?

  • converted mathematical expressions from SVG to inline MathML. This does not only improve readability in browsers, but also fixes broken math symbols when generating the PDF.
  • replace XSLTHL by highlight.js for better syntax highlighting
  • added fork me on github badge to website to visualize that one can easily contribute
  • enabled the Optimization Appendix. While it is not complete, it already provides some useful tips and maybe encourages contributions.
  • updated the Documentation build to work on Linux
  • added instructions how to Build the website/ PDF Docs

hopefully these changes will generate some momentum so this great Book gets extended again. As there were also non-cosmetical changes like the new Chapter I also tagged a 0.3.9 release.

I the process of the above work I found out that there is also a mirror of the original Book at http://alfonse.bitbucket.org/oldtut/. This one is however at the state of the 0.3.8 release, meaning it does not only misses the above changes but also some adjustment happened post 0.3.8 at bitbucket.

Categories: Graphics
madman2k

How to manually update a deb package from source

2014-03-15 12:03 UTC  by  madman2k
0
0

Probably everyone has encountered a package in Ubuntu which was not the newest released version while one for some reason needed the newest one. The first step is to search for a PPA with the desired version. But what if there is no such PPA or you want to build the version yourself? This is where this guide comes in. Note however that this is not aimed at ordinary users – you need some experience with programming/ compiling to successfully build a package.

Before you start

Before you start make sure that you have source packages enabled in your software sources.
Next you obviously need the upstream source tar-ball of the new program which should look something like <packagename><version>.tar.gz.
Download this tar-ball to a new directory <somedir> and extract it there.

Updating Package info

For the following commands I assume you are in the previously created directory <somedir>.

First we need to get the old version of the source package

apt-get source <packagename>

This will download and extract the old source package into <packagename><oldversion>.

Now we need some helper scripts to perform the upgrading as well as the build-time dependencies of the package

sudo apt-get install dpkg-dev devscripts fakeroot
sudo apt-get build-dep <packagename>

Next change into the extracted sources of the old package and update the packaging

cd <packagename>-<oldversion>
uupdate -v <newversion> ../<packagename>-<newversion>.tar.gz

# change into the extracted new package
cd ../<packagename>-<newversion>

# update version info
dch -l ~ppa -D $(lsb_release -sc)

For more information see the Debian New Maintainers Guide.

Building the program

To trigger a rebuild of the program simply execute

dpkg-buildpackage
Uploading your version to a PPA

To upload a package to a PPA you first need to sign it to prove that you are the author. To do this you have to execute the following in the <packagename><newversion> directory

debuild -S

Furthermore you need the upload tool dput to actually perform the uploading

sudo apt-get install dput

Now change to <somedir> and execute

dput ppa:<your_username>/<repository> <source.changes>

You can find more information at Launchpad.

Categories: News
madman2k

Secure Owncloud setup

2014-02-22 10:53 UTC  by  madman2k
0
0

While the Owncloud Manual suggests enabling SSL, it unfortunately does not go into detail how to get a secure setup. The core problem is that the default SSL settings of Apache are not sane as in they do not enforce strong encryption. Furthermore the used default certificate will not match your server name and produce errors in the browser.

Click to read 2210 more words
Categories: Articles