john quinn's blog

lamp on amazon ec2 shaping up nicely

john — Mon, 14 Apr 2008 21:02:46 -0700

recently i posted some encouraging performance benchmarks for drupal running on a variety of servers in amazon's elastic compute cloud. while the performance was encouraging, the suitability of this environment for running lamp stacks was not. ec2 had some fundamental issues including a lack of static ip addresses and no viable persistent storage mechanism.

amazon are quickly rectifying these problems, and recently announced elasic ip addresses; a "static" ip address that you own and can dynamically point at any of your instances.

today amazon indicated that persistent storage will soon be available.

backing up your xen domains

john — Sun, 10 Feb 2008 20:58:40 -0800

backups are boring, but we all know how important they are. backups can also be quite powerful when working with xen virtualization, since xen allows for convenient back-up and restore of entire systems.

i've recently been working on a flexible, general-purpose script enabling incremental backups of complete xen guests, optimized for secure, distributed environments; xenBackup. if you're working with xen, you might find it useful.

the xenBackup script leverages open-source components like ssh, rsync, and rdiff-backup to create a simple, efficient and functional solution.

lamp performance on the elastic compute cloud: benchmarking drupal on amazon ec2

john — Mon, 28 Jan 2008 05:54:48 -0800

amazon's elastic compute cloud, "ec2", provides a flexible and scalable hosting option for applications. while ec2 is not inherently suited for running application stacks with relational databases such as lamp, it does provide many advantages over traditional hosting solutions.

in this article we get a sense of lamp performance on ec2 by running a series of benchmarks on the drupal cms system. these benchmarks establish read throughput numbers for logged-in and logged-out users, for each of amazon's hardware classes.

we also look at op-code caching, and gauge it's performance benefit in cpu-bound lamp deployments.

load test your drupal application scalability with apache jmeter: part two

john — Fri, 18 Jan 2008 20:31:12 -0800

i recently posted an introductory article on using jmeter to load test your drupal application. if you've read this article and are curious about how to build a more sophisticated test that mimics realistic load on your site, read on.

the previous article showed you how to set up jmeter and create a basic test. to produce a more realistic test you should simulate "real world" use of your site. this typically involves simulating logged-in and logged-out users browsing and creating content. jmeter has some great functionality to help you do this.

load test your drupal application scalability with apache jmeter

john — Sun, 13 Jan 2008 21:48:06 -0800

there are many things that you can do to improve your drupal application's scalability, some of which we discussed in the recent scaling drupal - an open-source infrastructure for high-traffic drupal sites article.

when making scalability modifications to your system, it's important to quantify their effect, since some changes may have no effect or even decrease your scalability. the value of advertised scalability techniques often depends greatly on your particular application and network infrastructure, sometimes creating additional complexity with little benefit.

apache jmeter is a great tool to simulate load on your system and measure performance under that load. in this article, i demonstrate how to setup a testing environment, create a simple test and evaluate the results.

how to setup real-time email-notification for critical syslog events

john — Thu, 20 Dec 2007 07:11:10 -0800

a few weeks ago, i wrote a short article about the advantages of using syslog for all your logging needs. syslog is the standard logging solution for *nix platforms and integrates into virtually all application servers, network devices, and programming languages.

it is often important for system administrators to get real time notification of critical events. unfortunately, it isn't immediately obvious how to do this in the syslog framework. in this article i show you step-by-step how to do this.

supercharge your css code with m4

john — Wed, 19 Dec 2007 12:28:32 -0800

css has vastly improved the quality of html markup on the web. however, given its complexity, it has some astounding deficiencies.

one of the biggest problems is the lack of constants. how many times have you wanted to code something like this? light_grey = #CCC. instead you are forced to repeat #CCC in your css. this quickly creates difficult-to-maintain and difficult-to-read code.

an elegant solution to the problem is to use a general purpose preprocessor like m4. m4 gives you a full range of preprocessing capability, from simple constants to sophisticated macros.

using a guardian to ensure your lamp site is always up

john — Tue, 18 Dec 2007 11:02:55 -0800

to guarantee maximum uptime for your site, it's a good idea to periodically check the health of your system and restart failing components. you can use a simple program to do this automatically. i like to call this type of program, a "guardian".

clearly guardians shouldn't be used as a crutch for a badly configured system. used appropriately, however, they can decrease downtime due to unexpected events or administrator-error.

in this article, i describe how to implement, install and configure a guardian using a lightweight bash script. i go on to describe how to watch over your lamp install using this guardian. please note that all code and configurations have been tested on debian etch but should be useful for other *nix flavors with subtle modifications.

never lose your data again: backup remotely using rsync ssh and rdiff-backup

john — Wed, 05 Dec 2007 22:01:44 -0800

if you've ever lost precious data after a hard drive failure, you've probably learned your lesson and are now automatically backing up your system.

your treasured pictures, videos and documents may still be at risk. your computer could be stolen, destroyed by flood or fire or chopped into small pieces by a jealous ex-lover.

using a remote backup service is a good way to mitigate against this type of problem. for around $10 a month, you can find companies willing to store 10Gb of data for you. your data is usually accessible using a variety of methods, including rsync, vpn and ftp. to see some of these services, type remote backup rsync service into google.

in this article, i discuss using open source software to take advantage of these services in an efficient and secure manner, allowing the backup of large directories over a dsl-speed line while you sleep.

the fantastic four - drupals unofficial core

john — Wed, 28 Nov 2007 17:01:25 -0800

using the term "content management system" to describe the drupal cms understates it's full potential. i prefer to consider drupal a web-application development-system, particularly suitable for content-heavy projects.

what are the fantastic four?

drupal's application development potential is provided in large-part by a set of "core" modules that dovetail to provide an application platform that other modules and applications build on. these modules have become a de-facto standard: drupal's fantastic four. our superheros are cck, views, panels and cck field types and widgets. if you are considering using drupal to build a website of any sophistication, you can't overlook these.

scaling drupal step four - database segmentation using mysql proxy

john — Wed, 14 Nov 2007 18:30:33 -0800

if you've setup a clustered drupal deployment (see scaling drupal step three - using heartbeat to implement a redundant load balancer), a good next-step, is to scale your database tier.

in this article i discuss scaling the database tier up and out. i compare database optimization and different database clustering techniques. i go on to explore the idea of database segmentation as a possibility for moderate drupal scaling. as usual, my examples are for apache2, mysql5 and drupal5 on debian etch. see the scalability overview for related articles.

scaling drupal step one B - nfs vs rsync

john — Sun, 11 Nov 2007 06:02:59 -0800

i got some good feedback on my dedicated data server step towards scaling. kris buytaert in his everything is a freaking dns problem blog points out that nfs creates an unnecessary choke point. he may very well have a point.

having said that, i have run the suggested configuration in a multi-web-server, high-traffic production setting for 6 months without a glitch, and feedback on his blog shows gives example of other large sites doing the same thing. for even larger configurations, or if you just prefer, you might consider another method of synchronizing files between your web servers.

beef up your drupal security with apache mod_rewrite and SSH

john — Sat, 10 Nov 2007 06:22:29 -0800

if you felt a waft of cold air when you read the recent highly critical drupal security announcement on arbirary code execution using install.php, you were right. your bum was hanging squarely out of the window, and you should probably consider beefing up your security.

drupal's default exposure of files like install.php and cron.php present inherent security risks, for both denial-of-service and intrusion. combine this with critical administrative functionality available to the world, protected only by user defined passwords, broadcast over the internet in clear-text, and you've got potential for some real problems.

scaling drupal - an open-source infrastructure for high-traffic drupal sites

john — Sun, 28 Oct 2007 20:47:40 -0700

the authors of drupal have paid considerable attention to performance and scalability. consequently even a default install running on modest hardware can easily handle the demands of a small website. my four year old pc in my garage running a full lamp install, will happily serve up 50,000 page views in a day, providing solid end-user performance without breaking a sweat.

when the times comes for scalability. moving of of the garage

if you are lucky, eventually the time comes when you need to service more users than your system can handle. your initial steps should clearly focus on getting the most out of the built-in drupal optimization functionality, considering drupal performance modules, optimizing your php (including considering op-code caching) and working on database performance. John VanDyk and Matt Westgate have an excellent chapter on this subject in their new book, "pro drupal development"

once these steps are exhausted, inevitability you'll start looking at your hardware and network deployment.

scaling drupal step three - using heartbeat to implement a redundant load balancer

john — Sun, 21 Oct 2007 14:07:19 -0700

if you've setup a clustered drupal deployment (see scaling drupal step two - sticky load balancing with apache mod_proxy), a good next-step, is to cluster your load balancer.

one way to do this is to use heartbeat to provide instant failover to a redundant load balancer should your primary fail. while the method suggested below doesn't increase the loadbalancer scalability, which shouldn't be an issue for a reasonably sized deployment, it does increase your the redundancy. as usual, my examples are for apache2, mysql5 and drupal5 on debian etch. see the scalability overview for related articles.

scaling drupal step two - sticky load balancing with apache mod_proxy

john — Sun, 21 Oct 2007 13:55:33 -0700

if you've setup your drupal deployment with a separate database and web (drupal) server (see scaling drupal step one - a dedicated data server), a good next step, is to cluster your web servers. drupal generates a considerable load on the web server and can quickly become resource constrained there. having multiple web servers also increases the the redundancy of your deployment. as usual, my examples are for apache2, mysql5 and drupal5 on debian etch. see the scalability overview for related articles.

creating a xen bridging interface

john — Sat, 13 Oct 2007 10:56:47 -0700

in my previous blog, i go over a simple how-to for setting up xen on etch. in this configuration the xen guests are only visible to the xen-host, and any services on the xen-hosts must be accessed via port forwarding, tunneling etc.

for some applications, a bridging configuration works better. you can set this up as follows:

scaling drupal step one - a dedicated data server

john — Sat, 13 Oct 2007 08:26:29 -0700

if you've already installed drupal on a single node (see easy-peasy-lemon-squeezy drupal installation on linux), a good first step to scaling a drupal install is to create a dedicated data server. by dedicated data server i mean a server that hosts both the database and a fileshare for node attachments etc. this splits the database server load from the web server, and lays the groundwork for a clustered web server deployment. here's how you can do it. as usual, my examples are for apache2, mysql5 and drupal5 on debian etch. see the scalability overview for related articles.

easy-peasy-lemon-squeezy drupal installation on linux

john — Fri, 12 Oct 2007 19:23:17 -0700

installing drupal is pretty easy, but it's even easier if you have a step by step guide. i've written one that will produce a basic working configuration with drupal5 on debian etch with php5, mysql5 and apache2. it might be a help on other configurations too. see the scalability overview for related articles.

setting up syslog for distributed application logging

john — Tue, 09 Oct 2007 12:14:01 -0700

as you probably know, syslog is a *nix logging subsystem, typically used for system logging and kernel message trapping. you might not know that it's also an good tool for application logging, including large distributed systems. it has been around since the dawn of time and has very broad language, application and device support. here's a quick how-to to get you started, with debian etch examples.

how to use images in your drupal content

john — Sun, 07 Oct 2007 10:34:42 -0700

if you simply want to add image content to your drupal nodes etc, the array of options can be baffling. here's a quick overview of some options you might want to consider.

adding a javascript counter to your drupal forms

john — Sun, 30 Sep 2007 16:38:16 -0700

if you want to add a javascript counter (n characters remaining) to your drupal forms, you can do the following (jquery code based very heavily on the jquery tricks on reindel.com)

adding external javascript content to your drupal nodes or views

john — Sun, 30 Sep 2007 14:20:30 -0700

sometimes you want to include jquery / javascript code in content areas in your nodes, views etc. it's arguable that this code is better in managed in the filesystem (under source code control) than in your database-stored-node. here's how you might do it using a php include.

using jquery for in-place prompt text in your exposed drupal filters

john — Sun, 30 Sep 2007 10:58:43 -0700

when using exposed filters in drupal, it's sometimes nice to place the prompt text for a form field inside the form field itself e.g. for a very compact form. here's a way to do this using jquery.

setting up xen on your debian etch box

john — Sat, 29 Sep 2007 10:07:00 -0700

xen is a free software virtual machine monitor for IA-32, x86-64, IA-64 and PowerPC architectures. it runs on a host operating system and allows several guest operating systems to be run on top of the host on the same computer hardware at the same time.

there are many ways to setup xen, but i've put together a simple step-by-step guide to get a working xen system based on debian etch. easy as pie.

changing the date display in twiki from gmt

john — Sat, 22 Sep 2007 15:25:06 -0700

i know many people argue that your twiki install should display all dates in GMT/UTC, since the twiki is intended for international collaboration. well my home twiki4 install certainly is not, and if you're like me, you'll want to change it to something more sensible, without hacking up your install.

protecting your drupal site under development

john — Sat, 15 Sep 2007 13:37:08 -0700

while you are developing your user site, you probably want to protect it from prying eyes. you can do this using a drupal module or an apache configuration.