Understanding Load Average on Linux
One of the things that really confused me  back in the last millennium was understanding the results I would get from "top -c" or "uptime".  They showed load averages that seemed to make sense when they were low:  "0.32  looks like a low load. Is that 32 and I'm using 1/3 of the server capacity?"  It got more confusing when the number was something like 2 and the server seemed like it wasn't busy at all.  That made no sense because then wouldn't 2 mean 200%??
The load numbers on a Linux system are often confusing for people coming from Windows and other operating systems. The number often looks like a percentage, but it isn't.
When you evaluate the system load you have to know the number of CPUs your system has. Otherwise those numbers mean nothing.
To describe this in general terms as it was taught to me, I would first start with the concept that a CPU core can do 1 thing at a time. Knowing this, the number of items in the system run queue can start to make sense. The system queue is the jobs that are either being worked on, or about to be started.
In a 1 CPU system any number below 1 means that the system is keeping up with its tasks. It has 1 CPU and there is on average 1 task being worked on (or about to be started). In a 2 CPU/dual core system, if there is 1 item in the system queue, one or the other CPU is working on it or just about to grab it and you won't have any delay. If you are on a 16 core server and your load average is creeping up to 14, you are still not in bad shape because that's less than your CPU core count and all of the jobs are being worked on or are about to be started.
Now if your numbers are double your CPU count, then you are definitely overloaded and the system is falling behind. All of the CPUs are busy and they all have another job to work on after they are done with their current job.
After that, you need to watch out for a snowballing effect as the system starts to spend more and more time juggling its resources and less time processing the demand of the users.
Oh and why are there three numbers? Those are averages for different time periods. They are a  1 minute average, a 5 minute average and a 15 minute average.
( This explanation can get much more complicated. Â For example those 3 numbers are exponentially weighted moving averages (EMA or EWMA), meaning that in a 15 minute average, the most recent load measurements carry more weight than the older ones. So a load that was ramping up in that 15 minute period would produce a much higher average than a load that was inverse and decreasing at the exact same rate in that 15 minute period. But since I want to keep this explanation simple, I'm not going to even mention that.)
Hope that helps!
Understanding free memory in Linux
It used to worry me when I found that Linux was using almost all the memory available to a system. However all that worry was for naught. Linux is very good at memory management and making sure it has enough memory to do what it needs to do. You can run out of memory of course, but you are likely in better shape than you think you are.
If you run a command like "top -c" your server will likely tell you almost all the memory is used:
# top -c
top - 12:26:21 up 4 days, 3:09, 2 users, load average: 0.73, 0.58, 0.48
Tasks: 568 total, 1 running, 567 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 0.2%sy, 0.1%ni, 97.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8165824k total, 8064488k used, 101336k free, 83604k buffers
Swap: 6088624k total, 1338756k used, 4749868k free, 2445728k cached
Glancing at this result, you would think I only have 101mb free of 8gb here. However those numbers are as misleading as the load average on a multi-cpu server.
If you look at another command
# free -mt
total used free shared buffers cached
Mem: 7974 7921 53 0 27 2107
-/+ buffers/cache: 5786 2187
Swap: 5945 923 5022
Total: 13920 8844 5075
You get a similar misleading result, but you get to see the actual server condition too.
As you can see, the first free value is very low and that's what is concerning you.
However I want to draw your attention to the next line. That's really where you need to watch.
If you look at the buffers/cache line, you can see that used value is 5786mb and we have 2107mb in the free column. That free column is the Free + Cached + buffers (plus/minus rounding error of less than 2 Kbytes). That's really the line that you need to watch.
From that line we can tell that we have used 5.79gb out of the 7.97gb of total physical memory already used by programs. We can also see that we have 2.19gb of RAM that is in the cached pool that is available for usage.
As I mentioned before, Linux doesn't usually let memory go to waste. So you will watch that free number drop on the first line down to the double digits, but even then the cached value will be around 2gb. That means is we have roughly 2gb of memory available for programs right now. If a program needs more, it will pull it out of the cached memory pool and even after that, it will use the swap space before it is really out of memory and that is an additional 5gb.
To look at how much memory each program is using, I use this line:
# ps aux|head -1;ps aux | sort -nr -k 4 | head -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
102 4123 0.4 10.8 1196416 885840 ? Ssl Feb18 25:26 memcached [...]
mysql 17929 7.6 3.0 949372 248040 pts/3 Sl 06:05 23:32 /usr/libexec/mysqld [...]
root 7059 0.0 1.6 204360 138040 ? Ssl Feb18 0:28 /usr/sbin/clamd
apache 26931 1.2 0.8 225096 69956 ? S 11:06 0:05 /usr/bin/php-cgi
apache 26631 1.0 0.8 223304 66856 ? S 11:04 0:05 /usr/bin/php-cgi
apache 26458 1.5 0.8 223488 66824 ? S 11:03 0:09 /usr/bin/php-cgi
apache 23879 0.5 0.8 225068 68376 ? S 10:48 0:08 /usr/bin/php-cgi
root 26404 0.0 0.7 131956 58708 ? S Feb20 0:04 spamd child
root 24156 0.1 0.7 136320 63308 ? S 07:04 0:28 spamd child
apache 26937 2.5 0.7 221812 59788 ? S 11:06 0:11 /usr/bin/php-cgi
apache 26567 0.6 0.7 222416 61756 ? S 11:04 0:03 /usr/bin/php-cgi
apache 26405 0.0 0.7 222748 58228 ? S 11:03 0:00 /usr/bin/php-cgi
apache 23890 0.4 0.7 214040 57508 ? S 10:48 0:06 /usr/bin/php-cgi
apache 23851 0.1 0.7 221972 58596 ? S 10:48 0:01 /usr/bin/php-cgi
apache 17990 0.0 0.7 223916 58320 ? S 06:06 0:00 /usr/bin/php-cgi
apache 17164 0.1 0.7 215152 58956 ? S 10:13 0:04 /usr/bin/php-cgi
apache 14406 0.0 0.7 221164 63616 ? S Feb21 0:05 /usr/bin/php-cgi
root 7099 0.0 0.6 124312 49792 ? Ss Feb18 0:03 /usr/bin/spamd [...]
apache 26932 1.3 0.6 212336 52944 ? S 11:06 0:05 /usr/bin/php-cgi
apache 26628 2.2 0.6 213964 55164 ? S 11:04 0:11 /usr/bin/php-cgi
That shows you the 20 most memory intensive programs. Right now, on this server, the top to are memcached and mysqld - as it should be. Then there's a huge list of php-cgi instances prelaunched to handle an influx of connections. Also the spam checker coming up a few times. Most of the instances only take up ~220K, which isn't bad either. So from this, I can see that I have used a lot of memory in preparing many instances of php that are ready to go as connections come in. I also have APC installed and that is allowing the use of shared memory and that is reducing the overall footprint.
All in all, while I am showing a really low free memory value on this server, I know I actually have more memory available and already have a lot of existing memory taken up in preparation for when I get a much heavier load. As I speak, the server is dealing with 630 connections quite nicely.
BONUS TIP
To visually monitor memory usage, try this:
watch -n 1 -d free -mt
HOW TO: In PHP the MySQL Client API version doesn’t match the MySQL Server version
This is a fairly common situation. The short answer is that you usually don't need to fix anything. This is a non-issue. As long as your MySQL and MySQL Client have the same major version, you can and probably should just ignore the issue. There are no compatibility or performance issues involved.
This situation occurs when you use a package installer such as Yum to install a previously compiled version of PHP rather than compiling the version locally yourself. The package creator will have compiled PHP with a version the MySQL client that is compatible with that major release of MySQL.
To quote a 2008 article from the IUS Community project:
That said, I can tell you that in the last 4+ years of providing packages in this 'mixed' kind of fashion... we've never really come across any major issues of using php built against an older version of mysql... talking to a new version of mysql.
Now if you had to fix this, you could check to see if there is a specially compiled version of PHP with your particular MySQL client installed, but that's unlikely. You're more than likely going to need to recompile php from your command line. That slightly mismatched version is just the price you pay for convenience.
HOW TO: Determine which flavor or version of Linux is installed
If you are working with a Linux or Unix server, you may not even know which operating system release is on the server or even if it is a 32 bit or 64 bit machine. How do you even tell if your server is 32bit or 64bit? Usually this is not too terribly important. Most of the commands just work. However, it can become important when you want to install or upgrade the server software.
Fortunately, getting the OS version and machine information is JUST as simple..
cat /etc/*-release
That one line will work all versions that anyone who needs to read this article will ever run into.*
Debian: /etc/debian_version, /etc/debian_release (rare)
Fedora Core: /etc/fedora-release
Gentoo Linux: /etc/gentoo-release
Mandrake: /etc/mandrake-release
Mandriva/Mandrake Linux: /etc/mandriva-release, /etc/mandrake-release, /etc/mandakelinux-release
Novell Linux Desktop: /etc/nld-release
Red Hat: /etc/redhat-release
Solaris SPARC: /etc/release
Sun JDS: /etc/sun-release
SUSE Linux: /etc/SuSE-release, /etc/novell-release
SUSE Linux ES9: /etc/sles-release
Ubuntu Linux: /etc/lsb-release
This will produce results like:
CentOS release 5.5 (Final)
or
Red Hat Enterprise Linux Server release 5.4 (Tikanga)
The next thing you need to know is if the kernel you have installed is 64 bit ot 32 bit. You can do that via this command
cat /proc/cpuinfo
Here is an example of a 32bit kernel: (Notice the x86)
Linux version 2.6.18-164.el5PAE (mockbuild@x86-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Aug 18 15:59:11 EDT 2009
Here is an example of a 64bit kernel:(Notice the x64)
Linux version 2.6.18-028stab070.5 (root@rhel5-build-x64) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Fri Sep 17 15:37:23 MSD 2010
I hope this helps!
* NOTE:
If you have built your own server, there is a possility you have to run a slightly different command line as some version use *_version as the file name.
These flavours include:
Red Hat: /etc/redhat_version
Slackware: /etc/slackware-version
Knoppix: knoppix_version
So if the above line produced no results try:
cat /etc/*version
See more examples of these files here on Linux Mafia
How do you set the local timezone on a CentOS Linux server?
Though installations will vary, this is usually done via a file named timezone in the /etc directory.
By default, there's probably already a file there set to the utc timezone.
So if you do a
cat /etc/timezone
you will see
TZif2UTCTZif2UTC
UTC0
Fortunately you don't have to decipher all that. Your install probably has files for every timezone imaginable in the /usr/share/zoneinfo directory. You just have to create a symbolic link to one of those files. Then when Linux asks for the timezone, it returns the file you've linked to.
For example for the eastern time zone, EST, you could get away with something like this:
ln -sf /usr/share/zoneinfo/EST localtime
However, what would be even better is if you could specify a city near you that shares your timezone.
For example, I would chose cities from the America directory. Specifically I would choose New_York. So my timezone files link with
ln -sf /usr/share/zoneinfo/America/New_York localtime
The directories can be very specific... as we have some very specific people - like those crazy Indianans. People in the state of Indiana can be really whacky about timezones. So you can be even more specific like:
ln -sf /usr/share/zoneinfo/America/Indiana/Winamac localtime
In the end, you look around, chose the file that is best, remove the existing localtime file and create a link. Pretty simple really.
Determining what versions of WordPress you are hosting
If you host lots of different sites for people, one of the things you might want to know is what versions of WordPress each site is running.
WordPress stores the version number in a variable named $wp_version which is set in the file version.php.
With that information in hand, you can write a bash command that you run from your /home directory to display all of the WordPress versions you have on your server:
find . -name version.php -type f|xargs grep ^\$wp_version
This is one of the aliases I have in my .bashrc file.
Three helpful additions to your .bashrc
I just made a change to my .bashrc file and I thought I would share the tip. All of this is pretty basic stuff, but if you don't customize your Linux logins, this would be a good place to start.
For Microsoft people who don't know, .bashrc is in some ways like a combined config.sys and autoexec.bat file. If you don't know what an autoexec.bat file is, you totally missed the 80s dudes...
In a *nix environment, the rc at the end of a file name typically means that it is a "run control" file. Run Control files execute when a program starts. In this case, the program is bash - the command line interpreter/shell. Other programs look for rc files too. Because of this, you could have bunches of them in your home directory. The . at the front of the file name indicates that they are hidden from a normal directory listing. This way they don't clutter up your home.
I have lots of neat things in my .bashrc file that add functionality to my default CLI. I'll be sharing just three of those with you now.
The first is an alias: ebrc. When I type ebrc and press enter, I'm taken immediately into an editor with my .bashrc file open. You can think of an alias as a single line shortcut. It looks like this:
alias ebrc='vi ~/.bashrc'
As you can see, it just says "when I type 'ebrc', treat it like I really typed 'vi ~/.bashrc'".
The second thing I used tonight was the alias brc:
alias brc='. ~/.bashrc'
That executes the .bashrc file again, so that all of the changes I'd just made are loaded.
You might ask "Can't you just type all that out? You're not saving much time." Go ahead.. ask. I'll wait...
OK. The answer is Yes. So, it is important that you don't go overboard on this stuff. If you use aliases too much, you'll lose your familiarity with *nix and the skills to do your work on any other server. So proceed with caution. This stuff can be addictive and detrimental to your guru health.
Now with those two helper aliases in hand, I added the function I really wanted to include: 'upskel'. It takes a task I might otherwise put off and allows it to be completed in 7 keypresses. This is the perfect use case for a .bashrc function.
'upskel' takes the latest version of WordPress and places it into the cpanel skeleton directory that is used as the base for every new account created on my hosting service eHermits, Inc.. So, every time an update comes out for WordPress, I can spend 5 seconds to grab the latest and all new accounts I create will be safe and updated.
Unlike an alias, this is done through a function call. Functions allow the use of multiple lines and variables. Here is the call I just added:
function upskel() { cd /root/cpanel3-skel rm -R public_html rm latest.zip* wget http://wordpress.org/latest.zip unzip latest.zip mv wordpress public_html }
Technically I probably could have done that as an alias but a function is much easier to read with multiple lines involved.
As a bonus, here is a function that takes a variable:
function ewpc() { cd /home/$1*/ pwd sudo vi ./public_html/wp-config.php }
Can you tell me what it does?
Using find to copy specific files on Linux
I was faced with a weird copy command I wanted to do today; so I thought I would share.
How do you copy files in a directory tree to another directory?
I wanted to copy all of the mp3 files in a directory tree over to one specific target directory.
Initially I thought the command
cp -r *.mp3 /target/directory/
would work, but even though it specifies the --recursive option, it does not iterate the subdirectory looking for the mp3 files.
So I resorted to my every faithful companion: find -exec. It seems like this is one of the most useful tools in Linux. In this case, here is the command line I used:
find . -type f -name '*.mp3' -exec cp {} /targetdirectory/ \;
How to: Find files edited in the last day
The process is straight forwarded. There are several methods:
find . -mtime -1 \! -type d -exec ls -l {} \;
Or more simply
find . -type f -mtime -1
In my case, I wanted to do more. First since I was searching an SVN repository, I wasted to exclude all of the extra files that are touched by SVN. Additionally, the goal was to show which files contained a print_r function.
Here's what I came up with:
find . -path '*/.svn/*' -prune -o -type f -mtime -1 \
-exec echo '{}' \; -exec grep print_r {} \;
Hope that helps!
Getting the error ‘cannot move – to a subdirectory of itself’?
This error message always bothered me because it makes no sense when I get it.
Here's my scenario... I have a directory that is used as the base for all new accounts I sell on my servers. The path is; /root/cpanel3-skel. I always put WordPress in that directory and I want to keep the latest version of WP in that directory
[root@wiredtree ~]# rm -r wordpress latest.*
rm: cannot remove `wordpress': No such file or directory
rm: cannot remove `latest.*': No such file or directory
[root@wiredtree ~]# wget -q http://wordpress.org/latest.zip
[root@wiredtree ~]# unzip -q latest.zip
[root@wiredtree ~]# mv -f wordpress/* /root/cpanel3-skel/public_html/
mv: cannot move `wordpress/wp-admin' to a subdirectory of itself, `/root/cpanel3-skel/public_html/wp-admin'
mv: cannot move `wordpress/wp-content' to a subdirectory of itself, `/root/cpanel3-skel/public_html/wp-content'
mv: cannot move `wordpress/wp-includes' to a subdirectory of itself, `/root/cpanel3-skel/public_html/wp-includes'
WHAAAAT? Obviously it is not a subdirectory of itself...
Something strange is going on... We get a little more information if we try to do a move from one device to another. Take a look at this error message:
[root@wiredtree tmp]# mv -f wordpress/* /root/cpanel3-skel/public_html/
mv: inter-device move failed: `wordpress/wp-admin' to `/root/cpanel3-skel/public_html/wp-admin'; unable to remove target: Is a directory
mv: inter-device move failed: `wordpress/wp-content' to `/root/cpanel3-skel/public_html/wp-content'; unable to remove target: Is a directory
mv: inter-device move failed: `wordpress/wp-includes' to `/root/cpanel3-skel/public_html/wp-includes'; unable to remove target: Is a directory
This reveals the true source of the error message. In "mv --help", the explanation of "-f" is too simplified and says only:
-f, --force do not prompt before overwriting
In "cp --h" we closer,but not exact, explanation of the real process, and one that better matches the inter-device error message:
-f, --force if an existing destination file cannot be opened, remove it and try again
The final bit of information is in an Ubunto bug 71174 "Misleading error message with mv and existing directories" where they change the error message to be "mv: cannot move `a' to `b/a': Directory not empty".
So there you have it. You get that error message because the -f command tries first to remove the directory and can't because it contains one or more files. You don't get this message if the subdirectories are empty. The remove works fine on an empty directory but fails if there are files. The Linux core can't correctly handle this exception and throws up what is probably the last error message in a switch/case statement "cannot move - to a subdirectory of itself".
So what do you do about it? Well you can either empty the destination directory first, or you can copy the files and then delete the source directory. I chose the latter option and run this from the root folder:
rm -r wordpress latest.*
wget -q http://wordpress.org/latest.zip
unzip -q latest.zip
cp -rf wordpress/* cpanel3-skel/public_html/
rm -r wordpress latest.*
Cool? Hope that helps someone...