BluWiki:Carolyn
From BluWiki
Specs
Hardware:
- Processor: Intel Pentium 4 at 2.8GHz w/ HYPERTHREADING /1MB Cache, 800MHz FSB
- Memory: 2 GB DDR2, PC4200 533Mhz, ECC
- Hard drive:
- 2 160GB drives, Serial ATA, 7.2K RPM in Raid 1 configuration
- NIC: Gigabit Ethernet
Software:
- Gentoo Linux
- MySQL 4.1.19
- accelerated with Turck MMCache
All http is processed through Lighttpd on BluWiki:Ashelley
Updates & Notes
These are notes that I mainly keep for my own reference later - so I can retrace my steps if need be. They're posted here because they might be of interest to the BluWiki community as well!
April 30th, 2008 @ 10:13AM (EDT) - Completed server transition
The new BluWiki:Ashelley server is processing all HTTP requests via Lighttpd. Carolyn will now only serve as a msyql database. --Sam Odio 18:37, 30 April 2008 (EDT)
January 31st, 2007 @ 7:15PM (EST) - ANOTHER degraded array
For some reason, one of BluWiki's drives has failed. It's the same drive that failed last March. I guess I'll be taking the sever down later tonight to replace it.
November 12th, 2007 @ 8:56PM (EST) - Updated Mysql & Memcached due to OOM
carolyn just OOMed... so I tried adjusting some of the mysql variables to be more conservative: vim /etc/mysql/my.cnf:
thread_cache_size = 24 table_cache = 256
vim /etc/conf.d/memcached:
MEMUSAGE="256"
June 10th, 2007 @ 4:34 PM (PST) - Updated Mysql & Memcached log files
vim /etc/mysql/my.cnf:
key_buffer = 128M thread_cache_size = 70 table_cache = 512 sort_buffer_size = 1M read_buffer_size = 1M read_rnd_buffer_size = 768K thread_concurrency = 4 query_cache_size = 16M tmp_table_size = 32M max_heap_table_size = 32M
(used this page and this page as a template)
vim /etc/conf.d/memcached:
# $Header: /var/cvsroot/gentoo-x86/net-misc/memcached/files/1.1.12/conf,v 1.3 2005/12/29 11:55:01 lisa Exp $
MEMCACHED_BINARY="/usr/bin/memcached"
#Specify memory usage in megabytes (do not use letters)
#64MB is default
MEMUSAGE="768"
#User to run as
MEMCACHED_RUNAS="memcached"
#Specify maximum number of concurrent connections
#1024 is default
MAXCONN="1024"
#Listen for connections on what address?
LISTENON="127.0.0.1"
#Listen for connections on what port?
PORT="11000"
#PID file location
PIDFILE="/var/run/memcached/memcached-${PORT}.pid"
#Other Options
MISC_OPTS=""
Also:
$rc-update add memcached deafult
Note to self: after reboot, check that memcached's running with appropriate options:
$ps aux | grep mem 102 5711 0.0 0.7 17500 16240 ? Ss 19:02 0:01 /usr/bin/memcached -d -p 11000 -l 127.0.0.1 -m 1024 -c 1024 -u memcached -P /var/run/memcached/memcached-11000.pid
Also, run the status checks here: http://www.ibm.com/developerworks/linux/library/l-tune-lamp-3.html
April 13th, 2007 @ 4:28 PM (EST) - Installed Denyhosts
Per the instructions here: http://thinkhole.org/wp/2006/10/30/five-steps-to-a-more-secure-ssh/ and here: http://denyhosts.sourceforge.net/faq.html
Also switched the SSH port back to 22 in /etc/ssh/sshd_config
March 13th, 2007 @ 2:41 AM (EST) - Server Downtime
carolyn was offline for about 2 hours today (from approx. 3PM to 5PM EST), due to ISP troubles. From what they tell me, their entire Virginia customer base was down. --Sam Odio 02:43, 13 March 2007 (EDT)
March 10th, 2007 @ 8:18 PM (EST) - RAID Back up
I was able to shut down the server and remove the drive. Testing it with GRC's excellent SpinRite utility in another machine showed no physical damage. To add the drive back into the raid array, I first re-created the partition table:
sfdisk -d /dev/sda | sfdisk /dev/sdb
I then added each partition back into its respective array:
mdadm --manage /dev/md1 --add /dev/sdb1 mdadm --manage /dev/md2 --add /dev/sdb2 mdadm --manage /dev/md3 --add /dev/sdb3
March 7th, 2007 @ 9:01 AM (EST) - degraded array!
I guess it's a good thing I set up the disk monitoring, it appears that two of my arrays have become "degraded"! When I run $mdadm -D /dev/mdX, the state of md1 and md3 are as follows:
State : clean, degraded
I guess I'll have to run some drive tests to see if this is an actual device failure or just an anomaly. --Sam Odio 09:06, 8 March 2007 (EST)
March 7th, 2007 @ 8:35 AM (EST) - Hardware upgrades
I've made a few hardware changes to carolyn, which I've forgotten to post here. First I upgraded the RAM from 512 MB of PC 3200 to 2 GB of PC4200/DDR2. At some point I want to check into MySQL & MMCache to make sure they're appropriately utilizing the increase in resources, since I probably configured it for 512MB.
Also, I upgraded the system to RAID 1, using this tutorial. I've been able to watch the disk/raid stats using the watch command: "$watch -n .1 cat /proc/mdstat" - although I haven't really been doing this. Today I configured mdadm to email me whenever there's a device failure. To do this, edit /etc/mdadm.conf and add the line:
#Change root@example.com to your email address MAILADDR root@example.com
To test that the email notification works, run the following command:
mdadm -Fs1t
You should receive an email from mdadm at the supplied email address. If it works, you can then add mdadm to Gentoo's startup:
rc-update add mdadm default /etc/init.d/mdadm start
More detailed instructions are here.
March 7th, 2007 @ 8:18 AM (EST) APC Cache installation & benchmarking
I've installed the APC package for PHP caching, so that all PHP scripts aren't recompiled on each page load. APC was remarkably easy to install, simply execute the command pecl install apc and add "extension=apc.so" to the php.ini file.
The average page loaded in 77.073 ms with APC cache, which is an improvement of 65% on the original load time of 219.135 ms.
Without APC Cache:
$ab2 -n1000 http://bluwiki.com/go/Main_Page/ This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0 Server Software: Apache Server Hostname: bluwiki.com Server Port: 80 Document Path: /go/Main_Page// Document Length: 11489 bytes Concurrency Level: 1 Time taken for tests: 219.134777 seconds Complete requests: 1000 Failed requests: 111 (Connect: 0, Length: 111, Exceptions: 0) Write errors: 0 Total transferred: 11868874 bytes HTML transferred: 11488874 bytes Requests per second: 4.56 [#/sec] (mean) Time per request: 219.135 [ms] (mean) Time per request: 219.135 [ms] (mean, across all concurrent requests) Transfer rate: 52.89 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 203 218 44.9 205 832 Waiting: 192 206 43.6 194 819 Total: 203 218 44.9 205 832
With APC Cache:
$ab2 -n1000 http://bluwiki.com/go/Main_Page/ This is ApacheBench, Version 2.0.41-dev <$Revision: 1.121.2.12 $> apache-2.0 Server Software: Apache Server Hostname: bluwiki.com Server Port: 80 Document Path: /go/Main_Page// Document Length: 11489 bytes Concurrency Level: 1 Time taken for tests: 77.73395 seconds Complete requests: 1000 Failed requests: 108 (Connect: 0, Length: 108, Exceptions: 0) Write errors: 0 Total transferred: 11868875 bytes HTML transferred: 11488875 bytes Requests per second: 12.97 [#/sec] (mean) Time per request: 77.073 [ms] (mean) Time per request: 77.073 [ms] (mean, across all concurrent requests) Transfer rate: 150.38 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.0 0 0 Processing: 73 76 4.0 76 116 Waiting: 70 72 3.4 71 112 Total: 73 76 4.0 76 116 Percentage of the requests served within a certain time (ms) 50% 76 66% 76 75% 76 80% 76 90% 77 95% 78 98% 84 99% 99 100% 116 (longest request)
March 6th, 2007 - mod_deflate
I've configured Apache2 to use mod_deflate, which should compress all HTML/text documents served up on carolyn. I configured mod_deflate by adding the following lines to /etc/apache2/httpd.conf:
### The mod_deflate module provides the DEFLATE output filter.
### It allows output from your server to be compressed before
### being sent to the client over the network.
###
<IfModule mod_deflate.c>
# Insert filter
SetOutputFilter DEFLATE
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images, or PDF
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|pdf)$ no-gzip dont-vary
# Don't compress executibles or already compressed files
SetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-vary
# Make sure proxies don't deliver the wrong content
Header append Vary User-Agent env=!dont-vary
DeflateFilterNote Input input_info
DeflateFilterNote Output output_info
DeflateFilterNote Ratio ratio_info
LogFormat '"%r" %{output_info}n/%{input_info}n (%{ratio_info}n%%)' deflate
CustomLog /var/log/apache2/deflate_log deflate
</IfModule>
Dec. 2nd @ 1:00 AM (EST) - Any interst in other wiki engines?
I'm thinking about installing other Wiki engines besides MediaWiki (PHPWikik, etc.) on carolyn - but would like to gauge interest first. If someone is craving another open source engine, please leave me a note.
Nov. 24th @ 3:30 PM (EST) - Server's online!
Sever's live! Finially, after over 24 hours (since Noon on the 23rd), the server is back online. Looks like Keith got us past the subnet's firewall. --Sam Odio 15:37, 24 November 2005 (EST)
Nov. 24th @ 3:30 AM (EST) - downtime continues :(
Sever still down... It appears an improperly configured firewall is preventing the majority of the internet from accessing carolyn. Unfortunately, I do not have access to the firewall. Hopefully Keith (the guy who switched the subnet & controls the firewall) will be able to get to it soon. --Sam Odio 03:31, 24 November 2005 (EST)
Nov. 23rd @ 5:00 PM (EST) - Downtime notice
The server was switched over to a new subnet at 12:00 Noon. Since I was asleep after pulling an all-nighter (see below) - nobody was in the office to re-configure the server's network address. I just woke up, and the server now should be back online (for good!). Sorry for the downtime! --Sam Odio 23:59, 23 November 2005 (EST)
Nov. 23rd @ 6:30 AM (EST) - carolyn back online
After working through the night (literally) - carolyn is now installed and hosting BluWiki. Talk to me if there are any problems. --Sam Odio 06:32, 23 November 2005 (EST)
History
carolyn is a relatively new Dell PowerEdge server. It was the first server to host http://www.BluWiki.com using the Gentoo operating system. It also hosted http://www.odioworks.com and http://www.OnTheGrounds.com. The server performed exceedingly well, until an insecure password allowed a hacker to break in and use it as a spam relay.
Because of this break-in, the server was immediately taken offline, and all web hosting fell upon the shoulders of our backup server, Indigo, an old 1 Ghz Duron. She has performed well enough; however occasional freezes mean a replacement is needed.
I have now completely reconstructed carolyn (this time with the debian OS). (Went back to gentoo, an old favorite). I have also installed Turck MMCache, which will significantly increase the performance of BluWiki.
