HOW TO: Take a quick glance at errors from a project.
PHP records all errors to a file named error_log
I needed to look at the status of a project on two servers to see what errors were being thrown.
This command line shows the top 20 errors and how often they occurred within the last 1000 errors..
tail -1000 error_log | awk -F ']' '{print $4}'| sort | uniq -c | sort -nr | head -20
Depending upon the load on your server, that could report on the errors in the last half hour, or across several days. In either case, it will give you a quick idea on where you should clean your code of warnings and notices and where you need to do some fixing... Just remember there's no time line involved in the report. So the error messages could be from something already fixed.
HOW TO: Find the IP address of a domain from the local server before propagation
Have you ever wanted to know the ip address assigned to a domain according to your local server? I had this problem. I needed to get an IP address of a domain before the domain propagated. I was writing a custom cpanel postwwwacct script. For some very, very odd reason, cpanel will tell you IF you have a dedicated ip address for the domain, but not what it is. Fortunately postwwwacct occurs after the bind of the ip address.
I thought I would have to use a grep and awk to parse the IP address out of the .db file in /var/named but fortunately it occurred to me that I could just do a simple dig call within my script and get the value back. That's straight forward and returns only the information I want to have.
So here is a perl script that does this:
#!/usr/bin/perl $ns='ns1.example.com'; $domain='wahoo.com'; $ip=`dig \@$ns $domain A +short`; if ( $ip == -1 ) { print "command failed: $!\n"; } else { printf "The result is: " . $ip; }
You will of course have to replace ns1.example.com with the url for the name server running on your server.
The backticks (``) indicate you need to run a command line and return the stdout results. (And yes, I think backticks sound gross but that's what they are called.)
Also note the @ is a special character and must be escaped with a \.
Hope this helps!
HOW TO: Dump or Backup all MySQL databases to separate files
I needed to transfer about 40 databases from a new clients server over to my hosting platform.
When doing a couple database exports, I might use PHPMyAdmin to do the export. Heck, it's convenient because I usually want to look around at the same time. For doing a straight export, mysqldump is a great program that is even easier than phpmyadmin. There's really nothing to it! (NOTE: For large databases with a LARGE number of tables you need to add --skip-lock-tables)
mysqldump --all-databases > dbdump.sql
If you want to get fancy, you can even compress the file
mysqldump --all-databases | gzip -9 > dbdump.sql.gz
But I wanted to go one step further. I wanted every database to have a separate file, compressed and correctly named.
I was shocked how simple it was to write. First I had to ask mysql for a list of the databases. Then I needed extract just the database name, removing the table borders. Then I needed to tell mysqldump to use each database name in the export source and destination.
In no time I had a 3 line script that was incredibly powerful.
I give you backupdbs.sh
#!/bin/bash for database in $(mysql -e "show databases"|awk -F " " '{print $1}'); do mysqldump $database | gzip -9 > $database.sql.gz done
Note that you may need to provide the -u and -p parameters (username and password).
That's all there is to it!
The answer to: How do you set directories to 755 and files to 644?
AKA: Choosing and setting safe file permissions for a WordPress install.
If you have ever watched a WordPress security presentation, you've heard the advice:
Generally speaking, directories should be 755 and files should be 644.
But the presenter never tells you how to change these settings for all the directories and files in the WordPress install. I'm guilty of the same thing in my presentation. After all, Bash doesn't go over well in front of general audiences.
But how do you change all the directories and files to have the settings that you want? You're certainly not going to go around and manually change all of the permissions file by file & directory by directory! And if you did, would you still be able to upload files and upgrade your plugins with those permissions? Probably not.
I thought I would share how I handle this on my servers. My method is not the most refined, and does have some minor issues, but it works and gets me 95% to where I should be.
I have this function built into my .bashrc file:
function defaultr()
{
find . -type d | xargs chmod -v 755
find . -type f | xargs chmod -v 644
find . -type d -iname uploads | xargs chmod -Rv 774
find . -type d -iname blogs.dir | xargs chmod -Rv 774
find . -type d -iname cache | xargs chmod -Rv 774
find . -type d -iname themes | xargs chmod -Rv 774
find . -type d -iname plugins | xargs chmod -Rv 774
find . -type d -iname upgrade | xargs chmod -Rv 774
}
It is meant to be run from a www or public_html directory with WordPress installed under it. This will follow the basic rule, and modify the special directories so that you can upload, upgrade and use a caching tool. Additionally, it works for WordPress with multi-site either enabled or disabled.
Now, if you wanted to have some real fun, you could add lock and unlock functions so that there can be no changes to your plugins and themes unless you want them to be changed and then your site IS secure.
function lock()
{
find . -type f -iname wp-config.php | xargs chmod -Rv 644
find . -type d -iname themes | xargs chmod -Rv 754
find . -type d -iname plugins | xargs chmod -Rv 754
find . -type d -iname upgrade | xargs chmod -Rv 754
}
function unlock()
{
find . -type f -iname wp-config.php | xargs chmod -Rv 774
find . -type d -iname themes | xargs chmod -Rv 774
find . -type d -iname plugins | xargs chmod -Rv 774
find . -type d -iname upgrade | xargs chmod -Rv 774
}
Of course that means you need to be confident and not panic when you see messages like "Cannot update/modify/create x" and remember to unlock things before making those changes, but largely, you can set and forget these things until you are ready to apply updates.
One final note. The numbers in these examples are suitable for many shared hosts. If you pay more than $10/month for hosting, chances are you will be able to changes those numbers to better suit your needs. When I lock the files down, I use 544 and sometimes lower.
In any case, this is a good baseline for your functions and will get the job done. If you have a similar/better technique, I'd love to hear it.
(BTW have you figured out where the flaws are?)
Addendums:
- I should mention that the settings here are verbose. You can change the -Rv to -Rcf or just -Rf and get the same results less space taken up.
- If you get an error like "chmod: missing operand after '744'" This is "normal". It just means that your install does not have one of the directories. Maybe you've never upgraded a plugin and the upgrade directory doesn't exist. Or you have never run multisite and blogs.dir does't exist. You can either remove these lines or create empty directories, if the errors bother you.
- I've added echo lines so I can determine which commands create errors
In Linux, how do you unzip all the zip files in a directory?
This is a simple issue that trips me up every now and then. Let's say you have a directory of a couple dozen themes all in zip files. It would be a pain to type "unzip filename.zip" for every single one, but when you do an "unzip *.zip" you get something like:
[root@hosting ~]# unzip *.zip
Archive: file1.zip
caution: filename not matched: file2.zip
caution: filename not matched: file3.zip
caution: filename not matched: file4.zip
You see the problem is that as command line runs, the * is processed redundantly via both unzip and the *nix CLI. In the end, it tries to extract from the first file it finds, all of the remaining files in the directory by name.
The trick to fix this is simple. Escape the * with a back slash when making the call. Then the * is passed as a character to unzip and nothing more. Like so:
[root@hosting ~]# unzip \*.zip
Archive: file1.zipinflating: ...
Escaping symbols and aliases is important in other circumstances as well. Every now and then a Linux command simply won't work because the CLI has replaced text mid command with other text. I once couldn't FTP in somewhere because the password was replaced before FTP could actually run. Escaping the symbols in the password involved would have allowed the connection to work..
Adding a line to /etc/hosts via bash
It is easy to append a line to a file in BASH. I wrote a short script this morning that I can use to add a single line to my servers /etc/hosts pointing any domain I want at the loopback IP address of 127.0.01.
So I thought I would share..
#!/bin/bash SUCCESS=0 # All good programmers use Constants domain=example.com # Change this to meet your needs needle=www.$domain # Fortunately padding & comments are ignored hostline="127.0.0.1 $domain www.$domain" filename=/etc/hosts # Determine if the line already exists in /etc/hosts echo "Calling Grep" grep -q "$needle" "$filename" # -q is for quiet. Shhh... # Grep's return error code can then be checked. No error=success if [ $? -eq $SUCCESS ] then echo "$needle found in $filename" else echo "$needle not found in $filename" # If the line wasn't found, add it using an echo append >> echo "$hostline" >> "$filename" echo "$hostline added to $filename" fi
Three helpful additions to your .bashrc
I just made a change to my .bashrc file and I thought I would share the tip. All of this is pretty basic stuff, but if you don't customize your Linux logins, this would be a good place to start.
For Microsoft people who don't know, .bashrc is in some ways like a combined config.sys and autoexec.bat file. If you don't know what an autoexec.bat file is, you totally missed the 80s dudes...
In a *nix environment, the rc at the end of a file name typically means that it is a "run control" file. Run Control files execute when a program starts. In this case, the program is bash - the command line interpreter/shell. Other programs look for rc files too. Because of this, you could have bunches of them in your home directory. The . at the front of the file name indicates that they are hidden from a normal directory listing. This way they don't clutter up your home.
I have lots of neat things in my .bashrc file that add functionality to my default CLI. I'll be sharing just three of those with you now.
The first is an alias: ebrc. When I type ebrc and press enter, I'm taken immediately into an editor with my .bashrc file open. You can think of an alias as a single line shortcut. It looks like this:
alias ebrc='vi ~/.bashrc'
As you can see, it just says "when I type 'ebrc', treat it like I really typed 'vi ~/.bashrc'".
The second thing I used tonight was the alias brc:
alias brc='. ~/.bashrc'
That executes the .bashrc file again, so that all of the changes I'd just made are loaded.
You might ask "Can't you just type all that out? You're not saving much time." Go ahead.. ask. I'll wait...
OK. The answer is Yes. So, it is important that you don't go overboard on this stuff. If you use aliases too much, you'll lose your familiarity with *nix and the skills to do your work on any other server. So proceed with caution. This stuff can be addictive and detrimental to your guru health.
Now with those two helper aliases in hand, I added the function I really wanted to include: 'upskel'. It takes a task I might otherwise put off and allows it to be completed in 7 keypresses. This is the perfect use case for a .bashrc function.
'upskel' takes the latest version of WordPress and places it into the cpanel skeleton directory that is used as the base for every new account created on my hosting service eHermits, Inc.. So, every time an update comes out for WordPress, I can spend 5 seconds to grab the latest and all new accounts I create will be safe and updated.
Unlike an alias, this is done through a function call. Functions allow the use of multiple lines and variables. Here is the call I just added:
function upskel() { cd /root/cpanel3-skel rm -R public_html rm latest.zip* wget http://wordpress.org/latest.zip unzip latest.zip mv wordpress public_html }
Technically I probably could have done that as an alias but a function is much easier to read with multiple lines involved.
As a bonus, here is a function that takes a variable:
function ewpc() { cd /home/$1*/ pwd sudo vi ./public_html/wp-config.php }
Can you tell me what it does?
What is wrong with this bash script?
It should say that test.zip is a zip file, but the if statments, which check for TAR in the file name, all return true... If I do this outside of an if statement, say at the shell prompt, it works correctly.
What is wrong with this statement:
[dos]
#$bash
ArchiveName="test.zip"
cur_file="test.zip"
echo "ArchiveName: $ArchiveName"
if [ $(echo $ArchiveName | grep "tar$" -i)=$ArchiveName ]
then
echo Test1: It is a tar file
else
echo Test1: It is a zip file
fi
if [ $(echo $ArchiveName | grep "tar.gz$" -i)=$ArchiveName ]
then
echo Test2: It is a tar file
else
echo Test2: It is a zip file
fi
if [ $(echo $ArchiveName | grep "tar$" -i)=$ArchiveName ] || [ $(echo "$ArchiveName" | grep "tar.gz$" -i)=$ArchiveName ]
then
echo Test3: It is a tar file
else
echo Test3: It is a zip file
fi
[/dos]
Is there a better way to do this?
EasyWPUpdate ver 2.0 RC 1 – Just in time for WordPress 2.0.7
Well I can't call it the 5 second upgrade script any more... Since adding full file backups, and compressed database backups, from Windows desktop, through manual log in and launch of the update script, it took me ~15 seconds to update an active blog with a couple dozen posts and log all of the results to a html log. I'm fairly certain I could type my password faster and shave off a few seconds. The script itself, which now shows start and stop times, only took 2 seconds to do its work. The rest was connect and login time. Sometimes it took a whole 30 seconds for the process to complete, web and server usage being what it is, but either way, wow. I should say that I used this last night with all of the options turned on, creating file backups AND gzipped backups AND database backups AND an HTML log file AND adding extra verbosity AND updating my 6 WordPress blogs and it took a full 8.5 minutes. I had to actually minimize the window to get it out of my way.... Between that and typing in the script name at the shell prompt, I was exhausted!
When I think of how long it used to take me to update just my wife's blog, I just have to shake my head. Each release was a many-night, if not many-week process till I had the spare time to concentrate on doing the whole thing right. And I had to look up the instructions on the database backup every time... I'm just so glad this script is done.
The basic functionality is now complete and I am calling this a RC 1 release. It still needs further testing (especially the MySQLDump stuff. Does everyone HAVE MySQLDump? Should I disable this feature by default?), but it is fairly stable now.
Here's the basic functionality
# You can use this program in several ways:
# * In the default mode to download the latest and greatest update,
# make an uncompressed copy of your files, makes a compressed backup
# of your database(s), distributes the file to any number of
# directories, and performs the web steps
# * Configure it to make a online copy of your files you use for easy
# recovery AND a compressed copy that you can download.
# * Add custom directories and backup MORE than just WP.
# * Configure it update from a local file each night and start with
# a clean blog every morning.
# * Use it as a nightly backup script by disabling all other steps
I've made about a gazzilion improvements and took the advice of a dozen or more reviewers out there. I think the new script is much improved. I'm really pleased with how well the database backup stuff works. I change into each blog directory, read all of the connection information from the wp-config.php file and use that to connect to the database. (I'm fairly certain this will work well for most servers, but I'm a little worried that *nix gurus will not be using TCP to connect to their databases and I have that hard coded. You gurus should let me know if this is an issue in the comments for this post!) I also then query the tables names from the specified database using the prefix specified for the blog. This means that this process will work for ALL versions of WP and will not grab non-WP stuff like vBulletin tables. It also means that it works just as well if you have 1 blog per DB or ALL blogs in 1 DB. It doesn't matter. The DB backup for Blog1 has ONLY the data for Blog1. That is better for security, size, time and opens a neat avenue for testers who want to restore their blog to a different database/databasename and then test a major upgrade running their full blog out of a different directory. I've structured the tarball backups to make this easy too. *SORRY* There I go into tech speak again, but it is neat stuff, that is normally totally hidden from view.
You can peruse the text version, EasyWPUpdate.txt, here: link
Like the new name? I think it is better. I put TCC in front of all of my plugins, but really there is no need here. And yes, the sample version has grown to 851 lines. That's not ALL code of course. It is heavily documented and includes some HTML that will give you a nice webpage log for you to peruse after the process is done. You can see a sample log here: link.
Now, I had deliberately made that last post very intimidating. I wanted people to be wary of the script. Now, I have much more confidence in its ability and quality. I've learned a lot in the last week. From using procedures, to the trap function, to sed and MySQLDump, to basic shell coding practices. It was all fun and you get the benifit. Especially because there are three versions of WP in the pipeline: 2.0.7 (Very, Very Soon), 2.0.8 (in the works), 2.1 (Very Soon).
So, I've made this post easier to read and the script easier to configure. I'll do a full document later, but here are the basic steps to install this script:
1. Use Telnet or Putty to connect to your website and log into the shell
2. Type the following line:
wget http://www.thecodecave.com/downloads/EasyWPUpdate
3. Type the following line:
chmod +x EasyWPUpdate
4. Use an editor to change the values in Step 1 and save it again.
5. Run the script by typing:
EasyWPUpdate
That's it. You will have just made backups of the files and database and updated all of your blogs. When 2.0.8 comes out, the process will be:
1. Log in
2. Type
EasyWPUpdate
And you are done.
Now, step 1 looks like this:
# ##################################################################
# Step 1. Tell the script where to find the blogs
# ##################################################################
# List all of your WordPress directories and urls here.
#
# Each Blog should have a BlogDir and a BlugURL.
# Each Blog should have its own number [1],[2],[3] etc
# Delete the ones you don't need.
#
BlogDir[1]='site1dir'
BlogURL[1]='www.example.com'BlogDir[2]='site2/news'
BlogURL[2]='www.site2.com/news'BlogDir[3]='wordpress'
BlogURL[3]='blog.site3.com'
That isn't that hard to change is it? Even in VI.
Some quick tips on editing the script
1. type
vi EasyWPUpdate
2. Hit i
3. Make your changes
4. Hit ESCAPE COLON W to save your changes (or skip this step to lose changes)
5. Hit ESCAPE COLON !Q to immediately quit
Also, if your root directory is accessible from the web, you might want to change the name of the script
mv EasyWPUpdate SomeSneakyName
to prevent unauthorized access.
If you ran the alpha 3 version of the script, you can copy and paste configuration over BUT!!!! you have to make the following changes:
The BlogDirs[] array has been renamed to BlogDir[]. Drop the “s” from all of those variables.
You should not need to copy the Common*Prefix variables over, but if you do, make sure to remove the trailing slash from the CommonRootPrefix variable.
I think that’s all you need to be aware of.
If you are a guru, please read through all 6 setup steps (and the rest too) there may be things you want to change.
I've also updated my Did That Help page and added a forum specficly for this script.
That might make discussions a little bit easier.
Well that's about it. Let me know how it works. I'd like to get some good testing in before 2.0.7 comes out. I'll also do some testing with updating to 2.1 so I am certain that will work well. I also need write instructions for the database recovery steps. The script has built in instructions if it blows up in the middle of updating the files. So, that is handled.
I'll leave you with the change history and credits section from the script. Enjoy!
# History:
# 01/AUG/2006 - BL - Created
# 21/DEC/2006 - BL - Added multiple blog arrays
# Added options at the top of the script
# 04/JAN/2007 - BL - Added File Backup routines
# Added web update
# Added tmp directory usage
# Added local source "freshen" option
# 11/JAN/2007 - BL - Added "steps" and further comments
# Added quotes around many vars to protect against spaces
# Changed TMPDIR-/tmp to TMPDIR:-/tmp
# Changed `pwd` != "$tmp" to `$pwd` != "$tmp"
# Added further error trapping around cd and cp routines
# Fixed file backup procedure, was adding extra layers
# Added ability to backup to tarball
# Added SQL backup procedure
# Fixed local file backup procedure
# Removed “Verbose” from cp to make messages clearer
# Added log to webpage for Joe.
# Fixed inconsistent use of trailing / in path variables
# Added status messages throughout
# Added recovery instructions in case of failure mid backup
# Added a list of directories to backup
# Added Credits section
#
# Credits - I want to thank all of the readers of TheCodeCave.com, for
# their testing of this script. I especially appreciated Michael, Maciek,
# Aaron and Joe for all of the helpful suggestions.
# A very special thanks goes out to goldfish on the FreeNode #sed channel
# who will be PayPaled a Cafe Voltaire tomorrow. I would have spent days
# figuring out the RegEx for the SED commands. Prec, also from #sed gave
# provided me with a working cr/lf stripper. For bash, lhunath, jp-_ and the
# whole crew at #bash on FreeNode gave great line by line suggestions.
# They basically gave it a full code review! None of this would have been
# possible without Advanced Bash-Scripting Guide. 20 days ago I didn't know
# what bash was. Now I've written a powerful script with features I've not
# seen anywhere else. If you have any questions about the code in this
# script, you'll find the answers here: http://www.tldp.org/LDP/abs/html/
#
5 Second WordPress Upgrade Script – Status & Question
THIS ARTICLE IS OUT DATED. Please see: http://www.TheCodeCave.com/EasyWPUpdate for the current release.
Well I've gotten some good feed back from all of you. And I want to say Thanks!
I do have a new version of the script that has some improvements.
- Fixed extra directory levels in the backup
- Improved use of quotes
- Local file location did not untar the file
- Removed "verbose" from cp to make messages clearer
- $CommonRootPrefix no longer requires trailing '/'
- Added comment "DO NOT PUT A SLASH AFTER ANY OF THESE VARIABLES"
- Added a list of directories to backup
- Changed long backup section into a loop of the directories
- Changed to tmp directory before performing wgets
- Added a few more update steps for forward compatiblity
- Added SQL backup code
- Added Zip of backups
I'm doing local testing on these changes and will probably release this more publicly Wednesday or Thursday.
In the mean time, I would like to ask for your help.
What is the most optimized version of this line that we can come up with:
grep "define('DB_NAME', '" wp-config.php|sed -e 's/define(.DB_NAME....//g'|sed -e 's/.); .. The name of the database//g'
That line, run from any blog directory, returns the DB_Name in a way I can pass it to MYSQLDump. It is neither, pretty, nor optimized, nor resiliant.
What is the most optimized way YOU can think of to write that bash statement?
EDIT:
SED problem is solved. Thanks to GoldFish on the #SED channel of FreeNode.