WoodenBits

Today I got to play with my newly rebuilt VPS slice with shiny new Ubuntu 7.10 on it.

My current setup is apache and mongrels behind it, but wanted to try some new ideas and frameworks that appeared in Railscape in last couple of months.

Switchpipe looked like an interesting way to solve memory problems on small 256M VPS, where I run 3 mongrels and things feel quite a bit cramped with no more space for new rails apps.

I ran a simple performance tests using httperf hitting my server from another VPS on same local network, to eliminate artificial CPU and bandwidth related bottlenecks.

All tests used same "signup" page that is low on database usage, mostly rendering.

Participating frameworks:

• mongrel_rails 1.1.4
• thin 0.7.0
• ebb_rails 0.0.4
• switchpipe 1.0.4
• apache 2.2.4

Ok, here we go...

Naked Mongrel.

Good response rate, around 50 hits/s, but when rate of client requests went higher, it started to leak memory like a sieve, and process size went from 54M to more than 140M, and stayed there after load was long gone. Performance seriously degraded as hit rate went above 100.

Naked Thin.

Even slightly better response rate around 58 hits/second, and no memory leaks! Performance also degraded as hit rate went above 100, but not as bad as in case of mongrel.

Naked Ebb

Well, ebb was able to serve test page at whooping 1.3 hits/second. Not quite expected from server that is all about performance, but understandable given young age of the framework. Hopefully this is caused by some obvious problem that is going to be fixed soon.

Switchpipe managing 1 mongrel or 1 thin

Response rate dropped to about 20, and reason is obvious: switchpipe has loaded 100% of one CPU core, and mongrel/thin slacked at around 20% usage. Good news - no more memory leaks from mongrel any more, it feels much better when protected from overload by somebody.

Switchpipe managing 3 mongrels or 3 thins

Response rate went up to 50, with all 3 mongrels loaded at 20% and switchpipe 100%. Server also degrades nicer on hight load.

Apache proxying to switchpipe managing 3 mongrels or 3 thins

Same as above - apache seems to not introduce new bottlenecks.

Apache working as load balancer in front of 1 mongrel or thin

Pretty much same performance as naked mongrel or thin ( ~50 hits/second), but mongrel does not leak memory anymore and seems to have better degradation behavior under high load than thin.

Apache as a load balancer in front of 2 or 3 mongrels

Best performance of all - around 95 hits/second which degraded to 75 hits/second as load increased. What was interesting is that it did not mattered, 2 or 3 mongrels, performance was same. Maybe limits on the CPU usage of VPS were kicking in, maybe it was because server was low on memory with 3 mongrels (30M f of free RAM, cache included), but there were no benefit in running third mongrel.

Conclusions

Based on this experiment I decided that for my purposes combination of apache, switchpipe and mongrel or thin servers would work the best. It allows me to run as many 'test' rails apps on the box as I please, working around memory issues.

Still, switchpipe seems to be not very suitable for any heavy duty production work. If one of my server is to be slashdotted, the first thing to do is to take its configuration out of switchpipe, and run cluster of 2 mongrels under apache directly.

Also it looks like performance of thin and mongrel is pretty much the same for typical rails page, but thin is behaving better if server is to be used in a standalone fashion.

Well, here it is: I have to install Ruby on Rails app on fresh slice from slicehost.com, and this time I will use CentOS. And I know nothing about CentOS, and last time I used RedHat was around 8 years ago...

First - a reference - I found almost all information that I needed here: http://bryan.squall.us/posts/65

Here is my slightly modified script:

ssh -l root SERVER.IP.OR.NAME

yum update
yum install man wget

yum groupinstall "Development Tools"
yum install postgresql-devel
yum install ruby ruby-devel rdoc
yum install ruby-devel

wget http://rubyforge.org/frs/download.php/20989/rubygems-0.9.4.tgz
tar xfvz rubygems-0.9.4.tgz
cd rubygems-0.9.4
ruby setup.rb
gem update --system
gem install rails --version 1.2.6
gem install mongrel
gem install ruby-postgres

adduser www
passwd www

Well, I thought that my new 256MB VPS is really cool, and immediately deployed 3 mongrel clusters, 2 servers in each there, all load balanced by more or less standard apache configuration. Results? The thing was crawling. There were not enough memory to keep buffers and memory cache, so overall performance was miserable.

I realized my mistake and cut down each of those clusters to only 1 server (I know, at this point they are not really clusters, but I decided to keep it this was because it makes integration with capistrano easier, and allows for easy and quick expansion later down the road).

That provided immediate relief, but still... there were still not enough memory. Especially after I stress tested some of those mongrels with httperf, my swap grew to 150M, and things did not looked too good. I looked around on internet, and on some blogs and forums there was some information that 256MB slice was good only for 2 mongrels. Well, I am happy to report that with some tweaking 3 mongrels can be run more or less happily, with some room to spare on 256MB VPS!

Here is what I did.

First step was to realize that even that in idle state apache was not taking too much of memory (4-5k of RSS per process, couple of processes initially), during high load there were 150 of those processes! That quickly ate all the available memory.

Long story short, I added following lines to the bottom of my /etc/apache2/httpd.conf

ServerSignature Off
ServerTokens Prod

StartServers 1
MinSpareServers 1
MaxSpareServers 2
MaxClients 50
MaxRequestsPerChild 500
KeepAliveTimeout 2

Most of this means: do not start more than 50 apache processes at once, restart them after they served 500 requests (to get rid on any possible memory leaks), and do not hold connection to client open after response is served for more that 2 seconds (default is 15).

Now, that helped a LOT. Those settings are described well in other places on internet, I will not repeat that here, but tuning them helped a lot, especially under load.

One more thing that I have done to conserve even more memory is tuning rails. There is a way to tell rails to NOT LOAD frameworks that re not used. Here is how to do it in environment.rb file:

# Skip frameworks you're not going to use (only works if using vendor/rails) config.frameworks -= [ :actionwebservice, :action_mailer ]

Unfortunately, that works only if project is using it's own rails deployment, and it will not work if system-wide rails installation is being used. But, I wanted to use edge rails anyway, so it was a good excuse to do that as well.

Turning off web services in rails brought size on mongrel processes from 45MB RSS memory down to 30MB. Multiply 15MB by 3, and you will see that I gained 45MB of RAM this way!

This is the output of utility that displays amount of free memory on my system, that ran for couple days:

site ~ # free -m
             total       used       free     shared    buffers     cached
Mem:           254        251          3          0         11         40
-/+ buffers/cache:        198         56
Swap:          511          9        502

Not too shabby, right? ;)

So, here is is, my first post on this sparkling new blog of my own.

As I was setting blog up on fresh slice of slicehost, I found that there were number of instructions around how to do it, but none of them started from the very beginning, and they are mostly not based on Gentoo. So, I decided to write my own short guide on how to install initial rails environment on Gentoo.

I understand that people that dare to use Gentoo have good knowledge of basic building blocks, and personal preferences on them. This guide is just that, guide,not too detailed, without too many words to distract, and by no means foolproof. If you do not know what is Gentoo, you might be better off using some easier distribution, like Ubuntu. If you know and like Gentoo, I have to tell you that it is nice to use it on slicehost, with all the burstable raw CPU power.

Lets get started.

First, of course, buy a slice from slicehost ;) I got 256M one, and so far I liked it.

ssh to slice by IP and change root password:

ssh -lroot xx.xx.xx.xx
passwd

Login to slicehost manager and create your DNS entries (unless somebody else manages them). Create only one entry per domain ( for example, one entry for "woodenbits.com", and another one for "suretask.com"), than create aliases within those domains ("www", "blog", "svn" for woodenbits.com etc).

Now, back to ssh.

Add USE flags to your system:

USE="apache2 ruby ssl mysql postgres sqlite3 imagemagick gif jpeg jpeg2k png tiff svg wmf xpm pdf truetype unicode"

Optional, but recommended step - instal screen utility and start it

emerge screen
screen -h 9999

Upgrade your gentoo with fresh code out there:

emerge portage
emerge --update --deep --newuse world

Last step takes looong time, better run t overnight. Detach from screen session by pressing "ctrl-a d", than log out of ssh.

Come back next morning, ssh back to yuor slice, reattach your screen session with

screen -x

Merge all config files (about 29 of them):

etc-update

Keep your hostname config, it seems to be OK to replace rhe rest of them with updated version automatically.

Just in case, close shells in your screen session, that will exit it, than perform soft reboot from slicehost manager. That takes just seconds.

Ssh in there again, start screen again, and lets continue:

emerge ruby rubygems sqlite --ask

If you are planning on using pound for load balancing, get it:

emerge openssl pound --ask

If pound is still masked, unmask it so that emerge dares to install it: nano /etc/portage/package.unmask Add line

~www-servers/pound-2.3.2

Lets unmask newer versions of apache 2.2 and postgres 8.2 and install it

site ~ # more /etc/portage/package.keywords 
~www-servers/pound-2.3.2
~net-www/apache-2.2.4
~app-admin/apache-tools-2.2.4
~dev-libs/apr-1.2.8
~dev-libs/apr-util-1.2.8
~dev-db/postgresql-8.2.4
~dev-db/libpq-8.2.4


site ~ # more /etc/portage/package.unmask   
net-www/apache
app-admin/apache-tools
dev-libs/apr
dev-libs/apr-util

emerge subversion postgresql --ask
emerge --config =postgresql-8.2.4-r1
emerge --config =dev-util/subversion-1.3.2-r4

Now, lets start with rubygems installations

gem update --system
gem install rails mongrel mongrel_cluster postgres sqlite3-ruby -y

Lets get capistrano, it is going to make our life easier down the road.

gem install termios capistrano

Again, this is optional task, but those utilities are sooooo useful.

emerge vim sudo slocate --ask
locate -s

Create your subversion users list and first user, than create svn repository.

htpasswd2 -c /var/svn/conf/svnusers USER_NAME
svnadmin create --fs-type fsfs /var/svn/repos/YOUR_SVN_ROOT
chown -R apache.apache repos

Now, configure your apache, there are some excellent guides out there, I will not cover it here - call me lazy if you wish ;)

Tell gentoo to load apache on startup:

rc-update add apache2 default
rc

At this point you should have functioning system, and you can start deploying you rails applications there. Have fun doing it!