WoodenBits

Well, I thought that my new 256MB VPS is really cool, and immediately deployed 3 mongrel clusters, 2 servers in each there, all load balanced by more or less standard apache configuration. Results? The thing was crawling. There were not enough memory to keep buffers and memory cache, so overall performance was miserable.

I realized my mistake and cut down each of those clusters to only 1 server (I know, at this point they are not really clusters, but I decided to keep it this was because it makes integration with capistrano easier, and allows for easy and quick expansion later down the road).

That provided immediate relief, but still... there were still not enough memory. Especially after I stress tested some of those mongrels with httperf, my swap grew to 150M, and things did not looked too good. I looked around on internet, and on some blogs and forums there was some information that 256MB slice was good only for 2 mongrels. Well, I am happy to report that with some tweaking 3 mongrels can be run more or less happily, with some room to spare on 256MB VPS!

Here is what I did.

First step was to realize that even that in idle state apache was not taking too much of memory (4-5k of RSS per process, couple of processes initially), during high load there were 150 of those processes! That quickly ate all the available memory.

Long story short, I added following lines to the bottom of my /etc/apache2/httpd.conf

ServerSignature Off
ServerTokens Prod

StartServers 1
MinSpareServers 1
MaxSpareServers 2
MaxClients 50
MaxRequestsPerChild 500
KeepAliveTimeout 2

Most of this means: do not start more than 50 apache processes at once, restart them after they served 500 requests (to get rid on any possible memory leaks), and do not hold connection to client open after response is served for more that 2 seconds (default is 15).

Now, that helped a LOT. Those settings are described well in other places on internet, I will not repeat that here, but tuning them helped a lot, especially under load.

One more thing that I have done to conserve even more memory is tuning rails. There is a way to tell rails to NOT LOAD frameworks that re not used. Here is how to do it in environment.rb file:

# Skip frameworks you're not going to use (only works if using vendor/rails) config.frameworks -= [ :actionwebservice, :action_mailer ]

Unfortunately, that works only if project is using it's own rails deployment, and it will not work if system-wide rails installation is being used. But, I wanted to use edge rails anyway, so it was a good excuse to do that as well.

Turning off web services in rails brought size on mongrel processes from 45MB RSS memory down to 30MB. Multiply 15MB by 3, and you will see that I gained 45MB of RAM this way!

This is the output of utility that displays amount of free memory on my system, that ran for couple days:

site ~ # free -m
             total       used       free     shared    buffers     cached
Mem:           254        251          3          0         11         40
-/+ buffers/cache:        198         56
Swap:          511          9        502

Not too shabby, right? ;)

So, here is is, my first post on this sparkling new blog of my own.

As I was setting blog up on fresh slice of slicehost, I found that there were number of instructions around how to do it, but none of them started from the very beginning, and they are mostly not based on Gentoo. So, I decided to write my own short guide on how to install initial rails environment on Gentoo.

I understand that people that dare to use Gentoo have good knowledge of basic building blocks, and personal preferences on them. This guide is just that, guide,not too detailed, without too many words to distract, and by no means foolproof. If you do not know what is Gentoo, you might be better off using some easier distribution, like Ubuntu. If you know and like Gentoo, I have to tell you that it is nice to use it on slicehost, with all the burstable raw CPU power.

Lets get started.

First, of course, buy a slice from slicehost ;) I got 256M one, and so far I liked it.

ssh to slice by IP and change root password:

ssh -lroot xx.xx.xx.xx
passwd

Login to slicehost manager and create your DNS entries (unless somebody else manages them). Create only one entry per domain ( for example, one entry for "woodenbits.com", and another one for "suretask.com"), than create aliases within those domains ("www", "blog", "svn" for woodenbits.com etc).

Now, back to ssh.

Add USE flags to your system:

USE="apache2 ruby ssl mysql postgres sqlite3 imagemagick gif jpeg jpeg2k png tiff svg wmf xpm pdf truetype unicode"

Optional, but recommended step - instal screen utility and start it

emerge screen
screen -h 9999

Upgrade your gentoo with fresh code out there:

emerge portage
emerge --update --deep --newuse world

Last step takes looong time, better run t overnight. Detach from screen session by pressing "ctrl-a d", than log out of ssh.

Come back next morning, ssh back to yuor slice, reattach your screen session with

screen -x

Merge all config files (about 29 of them):

etc-update

Keep your hostname config, it seems to be OK to replace rhe rest of them with updated version automatically.

Just in case, close shells in your screen session, that will exit it, than perform soft reboot from slicehost manager. That takes just seconds.

Ssh in there again, start screen again, and lets continue:

emerge ruby rubygems sqlite --ask

If you are planning on using pound for load balancing, get it:

emerge openssl pound --ask

If pound is still masked, unmask it so that emerge dares to install it: nano /etc/portage/package.unmask Add line

~www-servers/pound-2.3.2

Lets unmask newer versions of apache 2.2 and postgres 8.2 and install it

site ~ # more /etc/portage/package.keywords 
~www-servers/pound-2.3.2
~net-www/apache-2.2.4
~app-admin/apache-tools-2.2.4
~dev-libs/apr-1.2.8
~dev-libs/apr-util-1.2.8
~dev-db/postgresql-8.2.4
~dev-db/libpq-8.2.4


site ~ # more /etc/portage/package.unmask   
net-www/apache
app-admin/apache-tools
dev-libs/apr
dev-libs/apr-util

emerge subversion postgresql --ask
emerge --config =postgresql-8.2.4-r1
emerge --config =dev-util/subversion-1.3.2-r4

Now, lets start with rubygems installations

gem update --system
gem install rails mongrel mongrel_cluster postgres sqlite3-ruby -y

Lets get capistrano, it is going to make our life easier down the road.

gem install termios capistrano

Again, this is optional task, but those utilities are sooooo useful.

emerge vim sudo slocate --ask
locate -s

Create your subversion users list and first user, than create svn repository.

htpasswd2 -c /var/svn/conf/svnusers USER_NAME
svnadmin create --fs-type fsfs /var/svn/repos/YOUR_SVN_ROOT
chown -R apache.apache repos

Now, configure your apache, there are some excellent guides out there, I will not cover it here - call me lazy if you wish ;)

Tell gentoo to load apache on startup:

rc-update add apache2 default
rc

At this point you should have functioning system, and you can start deploying you rails applications there. Have fun doing it!