How to Fix: After upgrading WHMCS: Down for Maintenance An upgrade is currently in progress

I recently came across an issue I’d not seen before. After performing what I thought was a proper upgrade, I found that my support and sales portal was displaying the error:

Down for Maintenance (Err 2)
An upgrade is currently in progress… Please come back soon…

I found several difference references to this error on the web. The one detail that I was able to glean was that this error usually occurs when the files are updated, but the database is not. This was the case in my situation. It turns out that I caused my own problem. The key to the mystery is that the files to perform the database upgrade are actually part of the “install” directory.

The instructions for a full upgrade are  here: http://docs.whmcs.com/Upgrading

Basically  you just:

  1. Extract the files
  2. Rename the admin and cron directories as needed
  3. Copy those files over the existing installation
  4. Visit the site and follow the upgrade instructions, if any.

My problem was that #4  includes an error at the end that tells you the install directory still exists, delete it.  Being a guy who likes to optimize a process as much as possible, I was deleting the install directory before copying  in the files.  This process worked great 99% of the time.  However, eventually they changed the database structure from 5.2.13 to 5.2.14 or 5.2.15.  THEN the result was that the WHMCS software could not upgrade the database. and I got the wonderful WHMCS Down for Maintenance error.

So really the steps should be:

  1. Extract the files
  2. Rename the admin and cron directories as needed
  3. Copy those files over the existing installation
  4. Visit the site and follow the upgrade instructions, if any.
  5. Delete the install directory when prompted.

At that point, everything should work fine.   At least it did for my particular incarnation of the problem.  I hope it solves your issues too.

How to quickly update WHMCS on WiredTree hosting and others.

NOTE: This script has been updated to remove the premature deletion of the install directory.

Over the last month or two, WHMCS has been updated many times. This is a GOOD THING, but it can be annoying to go through the normal update process every week or two.

So, I created a small function that I’ve added to /etc/bashrc on my server.

It does the following:
1. Verifies the script is running from the right server.
2. Changes directories to the location where I have WHMCS installed.
3. Removes any files and folders from a previous update
4. Downloads the latest update from the password protected WiredTree repository
5. Unpacks it
6. Renames the admin directory to a new name for security purposes (you may need to do the cron dir too)
7. Changes the owner and group of the new files to be the one required for this website.
8. Copies in all of the new files, overwriting what is there.

So updates now take a few seconds rather than 20 distracting minutes. Maybe you will find this useful.

Here is the function I added to /etc/bashrc:

 

function upwhmcs(){
  curhostname=$(hostname -s)
  desiredhost='REPLACE_WITH_HOSTNAME'

  if [ "${curhostname}" == "$desiredhost" ]; then
    cd /home/REPLACE_WITH_DIRNAME
    rm -Rf whmcs
    rm -f whmcs_current.zip
    wget http://REPLACE_WITH_USERNAME:REPLACE_WITH_PASSWORD@whmcs.wiredtree.com/whmcs_current.zip
    unzip -o whmcs_current.zip
    cd whmcs
    mv admin REPLACE_WITH_ADMIN_DIR_NAME
    chown -R REPLACE_WITH_DIROWNER:REPLACE_WITH_DIROWNER .
    cp -Rf * ../www
    cd ../www
    pwd
    return
  fi
  echo "Wrong Server. Update skipped"
}

 

To use that listing you need to replace the following text
REPLACE_WITH_HOSTNAME – Replace this with what you get when you run ‘hostname -s’ from the command line.
REPLACE_WITH_DIRNAME – Replace this text with the directory of your whmcs installation. Maybe it is /home/whmcs/ or /home/hosting/ or /home/sales/
REPLACE_WITH_USERNAME – This should become the username that WiredTree gave you to login and get WHMCS updates
REPLACE_WITH_PASSWORD – This should become the password that WiredTree gave you to login and get WHMCS updates
REPLACE_WITH_ADMIN_DIR_NAME – This is the custom name of the admin directory on your server. If you just use admin, delete this mv line entirely.
REPLACE_WITH_DIROWNER – This is quite likely the same value as REPLACE_WITH_DIRNAME

This script assumes that you have WHMCS installed as its own website/subdomain. You can probably adjust it to your needs if you run it in a sub-directory. If you make this customization, send it back to me and I’ll add it to the article.

After the upgrade process, you must visit the site, follow the update steps and then go back and delete the “install” directory from the installation.  That should do it!

How To: List the lines that exist in one file and not another in Linux or Windows Command Line

This solution uses grep.

Grep is a search tool that exists by default in just about every Linux installation. You can also search Google for Windows Grep.

for a Demo, Here is FileA.txt

1. In Both
2. In Both
3. In Both
4. In File A Only
5. In Both
0. Out of Order In Both
-1.  Out of Order In A Only
In A only

Here is FileB.txt

1. In Both
2. In Both
3. In Both
4. In File B Only
5. In Both
0. Out of Order In Both
-1.  Out of Order In B Only
In B only

To show lines that only exist in File B and not in A. Do this:

# grep -v -xFf FileA.txt FileB.txt

4. In File B Only
-1.  Out of Order In B Only
In B only

To show lines that only exist in File A and not in B. Do this:

# grep -v -xFf FileB.txt FileA.txt

4. In File A Only
-1.  Out of Order In A Only
In A only

 

In windows, Grep you can drop the capital F from the command line. But it really shouldn’t matter.

The command line arguments are:

-v = Reverse the compare so you get what doesn’t match instead of what does

-x = Compare entire lines at a time instead of characters

-F = Do not parse the contents of each line for any mid-line regex commands

-f = Compare the specified files

 

Hope that helps someone!

How to fix: WordPress updates and plugin installs require FTP login but shouldn’t.

WordPress is generally very good about working under any server situation you put it in. It even has fall back code if the preferred server settings are not available.

The plugin and update installation code is one example of this. If WordPress doesn’t have the rights to directly update files, it will ask you for FTP login information.

However, there are times when you KNOW the permissions are correct to allow Apache to update files and WordPress doesn’t detect it and insists on FTP. This should be extremely rare. However if you ever do come across this circumstance you can force WordPress to override this behavior and use direct updates.

Just add these lines to wp-config.php:

define( ‘FS_METHOD’, ‘direct’ );
define( ‘FS_CHMOD_DIR’, 0777 );
define( ‘FS_CHMOD_FILE’, 0777 );

You can read more about these constants on the WordPress Codex.

How to: Split path and filename in MS Excel, Google Docs or OpenOffice Calc

I did a code review along with Brad Williams yesterday and today we combined our results.  The problem was, our spreadsheet columns didn’t match.  He’d combined path and filename into one column and I’d split them out into two columns. We decided the split path and filename was useful.  So I had to come up with  a quick set of formula to create the two extra columns, splitting path from filename.

For a  short time, I thought it wouldn’t be possible, and that my formula foo had  finally failed me.  The difficulty is of course that the find()  doesn’t have an option for returning the position of anything but the first occurrence of the search text.  So, you don’t have anyway to know where to split the string.  Ahhh, but lo and behold the substitute function can be used to replace the nth occurrence of a character or string.  If you replace the right duplicated character with something unique, you can come use find() to determine the right place do do the split.

The method I used was based upon formulas found on Excel Ticker in an article written by Mourad Louha.  It took me several reads through to grok the concept of his method. (Actually I came back to the article hours later to figure out how the formula worked – long after hacking the formulas to do what I wanted). So, I thought I would boil it down for you here.  Also his formula was based upon an unusual layout in his spreadsheet.  What I’ve listed here simply assume a 3 column spreadsheet (Folder, File, Full Path) with a row for the title.  If your third column (c2) has the concatenated/full path, just paste the right formula into the first and second columns (a2 and b2) in the second row and you are done.

Here is  Mourad’s method boiled down into simple steps:

1. Determine how many slashes there are.  Subtract the length of the string, with slashes removed, from the original length of the string.

2. Replace the LAST occurrence of the slash with unique text we can find (we substitute $$$ but anything unique could be used)

3.  We now use find()  to get the position of the unique text and use that as the location to split the string.

 

Open Office Calc/Google Spreadsheet Filename Split Formulas

Linux Formula for splitting the path off a filepath:
= LEFT($C2;FIND(“$$$”;SUBSTITUTE($C2;”/”;”$$$”;LEN($C2) – LEN(SUBSTITUTE($C2;”/”;””))))-1)

Linux Formula for splitting a file name off a filepath:
=RIGHT($C2;LEN($C2)-FIND(“$$$”;SUBSTITUTE($C2;”/”;”$$$”;LEN($C2)-LEN(SUBSTITUTE($C2;”/”;””)))))

Windows Formula for splitting the path off a filepath:
= LEFT($C2;FIND(“$$$”;SUBSTITUTE($C2;”\”;”$$$”;LEN($C2) – LEN(SUBSTITUTE($C2;”\”;””))))-1)

Windows Formula for splitting a file name off a filepath:
=RIGHT($C2;LEN($C2)-FIND(“$$$”;SUBSTITUTE($C2;”\”;”$$$”;LEN($C2)-LEN(SUBSTITUTE($C2;”\”;””)))))

 

Microsoft Excel Filename Split Formulas

Microsoft Exceluses commas instead of semicolons.

Linux Formula for splitting the path off a filepath:
= LEFT($C2,FIND(“$$$”,SUBSTITUTE($C2,”/”,”$$$”,LEN($C2) – LEN(SUBSTITUTE($C2,”/”,””))))-1)

Linux Formula for splitting a file name off a filepath:
=RIGHT($C2,LEN($C2)-FIND(“$$$”,SUBSTITUTE($C2,”/”,”$$$”,LEN($C2)-LEN(SUBSTITUTE($C2,”/”,””)))))

Windows Formula for splitting the path off a filepath:
= LEFT($C2,FIND(“$$$”,SUBSTITUTE($C2,”\”,”$$$”,LEN($C2) – LEN(SUBSTITUTE($C2,”\”,””))))-1)

Windows Formula for splitting a file name off a filepath:
=RIGHT($C2,LEN($C2)-FIND(“$$$”,SUBSTITUTE($C2,”\”,”$$$”,LEN($C2)-LEN(SUBSTITUTE($C2,”\”,””)))))

Understanding Load Average on Linux

One of the things that really confused me  back in the last millennium was understanding the results I would get from “top -c” or “uptime”.  They showed load averages that seemed to make sense when they were low:  “0.32  looks like a low load. Is that 32 and I’m using 1/3 of the server capacity?”  It got more confusing when the number was something like 2 and the server seemed like it wasn’t busy at all.  That made no sense because then wouldn’t 2 mean 200%??

The load numbers on a Linux system are often confusing for people coming from Windows and other operating systems. The number often looks like a percentage, but it isn’t.

When you evaluate the system load you have to know the number of CPUs your system has. Otherwise those numbers mean nothing.

To describe this in general terms as it was taught to me, I would first start with the concept that a CPU core can do 1 thing at a time. Knowing this, the number of items in the system run queue can start to make sense. The system queue is the jobs that are either being worked on, or about to be started.

In a 1 CPU system any number below 1 means that the system is keeping up with its tasks. It has 1 CPU and there is on average 1 task being worked on (or about to be started). In a 2 CPU/dual core system, if there is 1 item in the system queue, one or the other CPU is working on it or just about to grab it and you won’t have any delay. If you are on a 16 core server and your load average is creeping up to 14, you are still not in bad shape because that’s less than your CPU core count and all of the jobs are being worked on or are about to be started.

Now if your numbers are double your CPU count, then you are definitely overloaded and the system is falling behind. All of the CPUs are busy and they all have another job to work on after they are done with their current job.

After that, you need to watch out for a snowballing effect as the system starts to spend more and more time juggling its resources and less time processing the demand of the users.

Oh and why are there three numbers?  Those are averages for different time periods. They are a  1 minute average, a 5 minute average and a 15 minute average.

( This explanation can get much more complicated.  For example those 3 numbers are exponentially weighted moving averages (EMA or EWMA), meaning that in a 15 minute average, the most recent load measurements carry more weight than the older ones. So a load that was ramping up in that 15 minute period would produce a much higher average than a load that was inverse and decreasing at the exact same rate in that 15 minute period. But since I want to keep this explanation simple, I’m not going to even mention that.)

Hope that helps!

Understanding free memory in Linux

It used to worry me when I found that Linux was using almost all the memory available to a system. However all that worry was for naught. Linux is very good at memory management and making sure it has enough memory to do what it needs to do. You can run out of memory of course, but you are likely in better shape than you think you are.

If you run a command like “top -c” your server will likely tell you almost all the memory is used:


# top -c
top - 12:26:21 up 4 days, 3:09, 2 users, load average: 0.73, 0.58, 0.48
Tasks: 568 total, 1 running, 567 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.4%us, 0.2%sy, 0.1%ni, 97.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8165824k total, 8064488k used, 101336k free, 83604k buffers
Swap: 6088624k total, 1338756k used, 4749868k free, 2445728k cached

Glancing at this result, you would think I only have 101mb free of 8gb here. However those numbers are as misleading as the load average on a multi-cpu server.

If you look at another command

# free -mt
total used free shared buffers cached
Mem: 7974 7921 53 0 27 2107
-/+ buffers/cache: 5786 2187
Swap: 5945 923 5022
Total: 13920 8844 5075

You get a similar misleading result, but you get to see the actual server condition too.

As you can see, the first free value is very low and that’s what is concerning you.

However I want to draw your attention to the next line. That’s really where you need to watch.
If you look at the buffers/cache line, you can see that used value is 5786mb and we have 2107mb in the free column. That free column is the Free + Cached + buffers (plus/minus rounding error of less than 2 Kbytes). That’s really the line that you need to watch.

From that line we can tell that we have used 5.79gb out of the 7.97gb of total physical memory already used by programs. We can also see that we have 2.19gb of RAM that is in the cached pool that is available for usage.

As I mentioned before, Linux doesn’t usually let memory go to waste. So you will watch that free number drop on the first line down to the double digits, but even then the cached value will be around 2gb. That means is we have roughly 2gb of memory available for programs right now. If a program needs more, it will pull it out of the cached memory pool and even after that, it will use the swap space before it is really out of memory and that is an additional 5gb.

To look at how much memory each program is using, I use this line:

# ps aux|head -1;ps aux | sort -nr -k 4 | head -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
102 4123 0.4 10.8 1196416 885840 ? Ssl Feb18 25:26 memcached [...]
mysql 17929 7.6 3.0 949372 248040 pts/3 Sl 06:05 23:32 /usr/libexec/mysqld [...]
root 7059 0.0 1.6 204360 138040 ? Ssl Feb18 0:28 /usr/sbin/clamd
apache 26931 1.2 0.8 225096 69956 ? S 11:06 0:05 /usr/bin/php-cgi
apache 26631 1.0 0.8 223304 66856 ? S 11:04 0:05 /usr/bin/php-cgi
apache 26458 1.5 0.8 223488 66824 ? S 11:03 0:09 /usr/bin/php-cgi
apache 23879 0.5 0.8 225068 68376 ? S 10:48 0:08 /usr/bin/php-cgi
root 26404 0.0 0.7 131956 58708 ? S Feb20 0:04 spamd child
root 24156 0.1 0.7 136320 63308 ? S 07:04 0:28 spamd child
apache 26937 2.5 0.7 221812 59788 ? S 11:06 0:11 /usr/bin/php-cgi
apache 26567 0.6 0.7 222416 61756 ? S 11:04 0:03 /usr/bin/php-cgi
apache 26405 0.0 0.7 222748 58228 ? S 11:03 0:00 /usr/bin/php-cgi
apache 23890 0.4 0.7 214040 57508 ? S 10:48 0:06 /usr/bin/php-cgi
apache 23851 0.1 0.7 221972 58596 ? S 10:48 0:01 /usr/bin/php-cgi
apache 17990 0.0 0.7 223916 58320 ? S 06:06 0:00 /usr/bin/php-cgi
apache 17164 0.1 0.7 215152 58956 ? S 10:13 0:04 /usr/bin/php-cgi
apache 14406 0.0 0.7 221164 63616 ? S Feb21 0:05 /usr/bin/php-cgi
root 7099 0.0 0.6 124312 49792 ? Ss Feb18 0:03 /usr/bin/spamd [...]
apache 26932 1.3 0.6 212336 52944 ? S 11:06 0:05 /usr/bin/php-cgi
apache 26628 2.2 0.6 213964 55164 ? S 11:04 0:11 /usr/bin/php-cgi

That shows you the 20 most memory intensive programs. Right now, on this server, the top to are memcached and mysqld – as it should be. Then there’s a huge list of php-cgi instances prelaunched to handle an influx of connections. Also the spam checker coming up a few times. Most of the instances only take up ~220K, which isn’t bad either. So from this, I can see that I have used a lot of memory in preparing many instances of php that are ready to go as connections come in. I also have APC installed and that is allowing the use of shared memory and that is reducing the overall footprint.

All in all, while I am showing a really low free memory value on this server, I know I actually have more memory available and already have a lot of existing memory taken up in preparation for when I get a much heavier load. As I speak, the server is dealing with 630 connections quite nicely.

 

BONUS TIP

To visually monitor memory usage, try this:

watch -n 1 -d free -mt

HOW TO: In PHP the MySQL Client API version doesn’t match the MySQL Server version

This is a fairly common situation. The short answer is that you usually don’t need to fix anything. This is a non-issue.  As long as your MySQL and MySQL Client have the same major version, you can and probably should just ignore the issue. There are no compatibility or performance issues involved.

This situation occurs when you use a package installer such as Yum to install a previously compiled version of PHP rather than compiling the version locally yourself. The package creator will have compiled PHP with a version the MySQL client that is compatible with that major release of MySQL.

To quote a 2008 article from the IUS Community project:

That said, I can tell you that in the last 4+ years of providing packages in this ‘mixed’ kind of fashion… we’ve never really come across any major issues of using php built against an older version of mysql… talking to a new version of mysql.

Now if you had to fix this, you could check to see if there is a specially compiled version of PHP with your particular MySQL client installed, but that’s unlikely.  You’re more than likely going to need to recompile php from your command line.  That slightly mismatched version is just the price you pay for convenience.

Cold storage before my best ideas melt away…