Ever wondered why the MySQL command line tool don’t have any sort of progress information or status update? Well, you’re not alone if you do. Especially importing a GB sized dump file can take a long time, especially if the table engine was InnoDB.

There are some clever utilities which can be used to accomplish just that, by giving the user simple, yet useful information about the progress of their process. Let’s look at them.

Bar (Command Line Progress Bar)

The Bar utility, or in full terms Command Line Progress Bar can be downloaded from Sourceforge. If you are running Ubuntu, it’s as simple as running ‘sudo apt-get install bar’ to install it. You then simply pipe your MySQL import and get a nice status bar.

shell> bar -if=mysql_db_data.sql | mysql

Piper Viewer

To get a bit fancier, Piper Viewer can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion. Observe multiple instances working in tandem,  a visual indicator of relative throughput in a complex pipeline. Download pv here.

pv -cN gzip mysql_db_data.sql.gz | gzip -d | pv -cN mysql | mysql

Either utility will undoubtedly shed more light into your restoration processes, and let you know whether the process is still running if in doubt and wondering after an hour or more.

I’ve been asked many times how to delete records in multiple tables in MySQL. Here are couple of methods that should get anyone started.

Simple Inner Join Method

The simple inner join method uses commas with an implicit inner join like in the example below. Here we’re using an items and price table, where each table has an itemId field which is what we join them on.

DELETE i.*, p.*
FROM items i, price p
WHERE i.itemId = p.itemId
AND i.itemId = 101

Traditional Inner Join Method

In the traditional inner join we’re keeping the join statement clear of the where clause, but it works as well. Whatever personal preference you may have. Note that you can delete individual table records from one table only, if you require to do so.

DELETE i.*, p.*
FROM items i
INNER JOIN price p
ON i.itemId = p.itemId
WHERE i.itemId = 101

Delete records that are missing on the second table

Using a left join, we can also delete records that have no corresponding record in the joined table. The following query accomplishes that. In the example below, we delete all records in the items table that have no record in the price table.

DELETE i.*
FROM items i
LEFT JOIN price p
ON i.itemId = p.itemId
WHERE p.itemId IS NULL

I don’t remember how many times I was asked about an equivalent term of the infamous “NOLOCK” hint for mysql database server, hence I thought it was worth to write about it here. “WITH (NOLOCK)” is a transaction isolation levels that defines how data is available during an update, or with other words it is a property that defines at what point changes made by an update operation will become available in a row, table or database to other processes.

The official SQL standard defines four isolation levels:

READ COMMITTED
READ UNCOMMITTED
REPEATABLE READ
SERIALIZABLE

Oracle, SQL Server and MySQL support isolation levels. During an operation, the database engine places certain locks to maintain data integrity. Different types of locking apply to different databases (Oracle vs. MySQL), or table types (eg. MyISAM vs. InnoDB).

When WITH (NOLOCK) is used with SQL Server, the statement does not place a lock nor honor exclusive locks on table. The MySQL equivalent is READ UNCOMMITTED, also known as “dirty read” because it is the lowest level of isolation. If we specify a table hint then it will override the current default isolation level. MySQL default isolation level is REPEATABLE READ which means locks will be placed for each operation, but multiple connections can read data concurrently.

SQL Server WITH (NOLOCK) looks like this:

SELECT * FROM TABLE WITH (nolock)

To achieve the same with MySQL, we change the session isolation mode using the SET SESSION command.

SET SESSION TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;
SELECT * FROM TABLE_NAME ;
SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ ;

This statement will work similar to WITH (NOLOCK) i.e READ UNCOMMITTED data. We can also set the isolation level for all connections globally:

SET GLOBAL TRANSACTION ISOLATION LEVEL READ UNCOMMITTED ;

In addition, two system variables related to isolation also level exist in MySQL server:

SELECT @@global.tx_isolation; (global isolation level)
SELECT @@tx_isolation; (session isolation level)

Or set the isolation level inside a transaction:

SET TRANSACTION ISOLATION LEVEL REPEATABLE READ
GO

CouchDB is another offspring from the open-source, NoSQL, non-relational databases and is maintained under the Apache Foundation. It differs itself form the likes of MongoDB or Cassandra in that CouchDB is storing data in so called “documents” that are in JSON format, which can be hashes, lists, nested arrays and of course scalar values. This added complexity results in more powerful features, mainly to have a db that is not just a single key/value pair, but it comes at a price of speed reduction.

CouchDB can be a little bit of a pain to install, because it needs a few pre-requisites and they in turn have a few of their own quirks. This outline should help you get CouchDB with all it’s necessities installed. We’ve used MacOS, but you can substitute your OS where applicable.

The CouchDB source code and installer packages are downloadable here. As of this writing, version 0.11.2 is the latest stable version, with 1.0.1 just around the corner. You will also need Spider Monkey, Mozilla’s C implementation of JavaScript.

Installing SpiderMonkey

Once downloaded, extract the tarball and move into the sources folder:

tar -xzvf js-1.8.0-rc1.tar.gz

cd js/src

If you’re installing SpiderMonkey on a Mac, there is a small hack to the jsprf.c file. Adding the following two lines (in orange below) after line 60 should resolve the build issue:

60: #define VARARGS_ASSIGN(foo, bar) VA_COPY(foo,bar)
61: #elif defined(va_copy)
62: #define VARARGS_ASSIGN(foo, bar)
63: #elif defined(HAVE_VA_LIST_AS_ARRAY)

Now we should have no issue running make:

make BUILD_OPT=1 -f Makefile.ref

sudo sh

sudo make BUILD_OPT=1 JS_DIST=/usr/local -f Makefile.ref export

Installing ICU

ICU is a library for International Components for Unicode and CouchDB relies on it. Let’s download it from icu-project.org here. Once downloaded, extract the package and move into the sources folder and change some permissions before building the code:

tar -xzvf icu4c-4_4_1-src.tgz

cd icu/source

chmod +x runConfigureICU configure install-sh

./runConfigreICU -h (select the suffix for your OS and and supply it as argument to runConfigureICU, eg: ./runConfigureICU MacOS)

make

sudo make install

Installing Erlang

Erlang is the language used to build CouchDB, and you’ll have to install it if it’s not yet present. Download Erlang version 14A (latest as of this writing) from erlang.org here or use wget or curl as in the steps below.
cd /tmp
wget http://www.erlang.org/download/otp_src_R14A.tar.gz
tar -xzvf otp_src_R14A.tar.gz
cd otp_src_R14A
./configure –enable-hipe –enable-smp-support –enable-threads (–enable-darvin-universal for Mac)
make
sudo make install

Installing CouchDB

Now finally we’re ready to install CouchDB.

cd /tmp

wget http://www.apache.org/dyn/closer.cgi?path=/couchdb/0.11.2/apache-couchdb-0.11.2.tar.gz

tar -xzvf apache

cd apache-couchdb-0.11.2

./configure

make

sudo make install

sudo couchdb start

Testing our CouchDB installation

Besides the countless client libraries written in almost any language, we can simply use Telnet to create a database and do some basic data manipulation to test our installation.

telnet localhost 5984

If there are no errors and you get a response like below, you’re all set.

Trying 127.0.0.1…
Connected to localhost.
Escape character is ‘^]’.

A good introduction to CouchDB’s commands can be found here.

I’ve been asked the question “How can I return duplicate rows only from a MySQL db table” so many times already, that I’ve decided to post it here in a short article.

It is not something intuitive or readily available (at least it seems), but the solution is short and very simple.

While this query:

SELECT DISTINCT column1
FROM table1

gives us all records without the duplicates, this one returns only the duplicate ones:

SELECT DISTINCT column1
FROM table1
GROUP BY column1
HAVING COUNT(column1) > 1

And by increasing the having count, you can retrieve records with multiple occurrences.

Ever come across a situation, where you’d like to check all tables in a database and have them all repaired and optimized? My guess is yes.

In case you didn’t know, there is a helpful MySQL utility called mysqlcheck, available as of version 3.23.38. It does exactly what we need.

To check all tables in all databases for corruption and errors and also fix them in one go, this is your command:

mysqlcheck -u username -p password  –check –optimize –auto-repair –all-databases

mysqlcheck executes statements like CHECK TABLE, REPAIR TABLE, ANALYZE TABLE, and OPTIMIZE TABLE and chooses the best statements for any given operation and storage engine.

Note that the operations complete a lot faster if you can afford to to disable any external services, especially if your database is large.

These instructions lead you thru the installation of the latest LAMP Stack on MacOS X Snow Leopard.

1 – Download the installation image from MySQL website here. Then double-click to mount and open the disk image.

2 – Install MySQL Server by double clicking the package “mysql-5.1.*****.pkg” and follow the menu, accepting the default values, unless you want to change something and know exactly what you’re doing.

3 – Install MySQL Startup Item by double-clicking the package “MySQLStartupitem.pkg” and follow the menu.

4 – Install MySQL Preference Pane by double-clicking the file “MySQL.prefPane” and follow the menu. This item will simplify the management of your SQL Server. You can now use the “System Preferences” panel to start and stop the database server.

5 – Enable the php module in your apache config file. You might know that Snow Leopard already ships with Apache 2.2 and PHP 5.3, but it needs a couple of tweaks to make it work smoothly. So, open /etc/apache2/httpd.conf and search for “php5_module”. Remove the comment (#) in front of the line, save and close the file, then restart apache (sudo apachectl restart)

6 – Testing Apache/PHP. Let’s create an info file so we know what we got. Navigate to /Library/WebServer/Documents and create a file called phpinfo.php. Put the following line into the file, save and close. “<?php phpinfo(); ?>”. Now open your favorite browser and hit it (http://localhost/phpinfo.php). You should see a page like the one below, which shows you configuration details, modules enabled and other useful information.

7 – There is no php.ini out of the box (as you may have discovered by looking at the phpinfo page), but there is a sample file called “php.ini.default” which we’re going to use. Copy this file and save it as php.ini.

8 – Set the default time zone. As PHP 5.3 requires the timezone to be set, let’s do this inside php.ini, so we don’t have to worry about it anymore. You can always override it in your application’s bootstrap, if needed. Open /etc/php.ini and search for “[Date]“. Uncomment the line “date.timezone = ” and add your timezone. Check the php manual for acceptable values. (example: date.timezone = ‘America/Los Angeles’)

And that’s it. You should now have a fully functional LAMP stack with Apache, PHP and MySQL server.

1. Configure the Master

We will need to modify a file called my.cnf, which is the main configuration file for mysql. On most systems it’s located in /etc/ or /etc/mysql/ and it contains all important configuration data.

First, let’s ensure that networking is enabled and mysql listens on all, or at least the client’s IP addresses. We also have to tell mysql what file to write the logs to and from which databases to keep logging, so that the Slave can pick up the changes. And finally we need to assign a unique ID to the Master.

All this info is contained within the following lines in your my.cnf file. Please note that the position of these entries can be spread throughout the file, so you might have to search for each of them.

Restart the server. Then log into mysql and create a user with replication privileges:

The next 3 steps is to lock all tables on the database, take a backup and get the replication sequence ID. We’ll use the backup later on the Slave to establish the baseline, and tell it to start replication starting from the sequence ID.

If you’re performing this action on a high volume production system, I’d recommend to plan these next steps carefully, as the db will be locked for writing during the backup process.

Write down the “Position” number, it’s the sequence ID and the bin log file. Since the tables are locked, you’ll need to open a second terminal and log into mysql and perform the backup:

Once the backup is finished, unlock the tables in the first terminal.

Whew, we’re done with the Master. Wasn’t that hard, was it? Now on to the Slave.

2. Configure the Slave

You’ll have to copy the backup file to the Slave server and extract it (gzip -d). On the Slave, let’s first create the database and restore the dump we took from the Master.

After loging into mysql from the terminal:

Then we’ll go on to configure my.cnf on the Slave. As I stated earlier, the position of these entries can be spread throughout the file, so you might have to search for each of them or they might not even be present, so you have to add them.

3. setup parameters and start replication

Restart MySQL, login to mysql and stop the Slave. You now will need the info you wrote down during the “SHOW MASTER STATUS” command.

Example: CHANGE MASTER TO MASTER_HOST=’192.168.1.1′, MASTER_USER=’slave_user’, MASTER_PASSWORD=’1234′, MASTER_LOG_FILE=’mysql-bin.001′, MASTER_LOG_POS=256;

Finally, we start the Slave with the new settings and get replication started. Still from the terminal, logged in to mysql:

That’s all there is to it. There are commands like “SHOW SLAVE STATUS” that let’s you check if replication is working.

Update:

If you have problems getting “Slave_IO_Running” to “Yes” state, you might have to flush the tables on the slave, like this:

STOP SLAVE;

FLUSH TABLES WITH READ LOCK;

UNLOCK TABLES;

START SLAVE;

Memcache is traditionally used as a module inside server side scripts, such as PHP, ASP, ColdFusion and others. And it’s doing a terrific job, as long as it’s implemented correctly.

But if we look under the hood of the actual Memcache application, and I’m not talking about the PHP or ASP extension, but rather the executable that’s running as a daemon under linux, for example, it is a rather simple database like application running in memory. Now there are two basic actions that need to be performed to use it, one is writing info into memcache, the other is reading it. In a typical scenario, there are many reads for one write, that’s the whole point, isn’t it. But what if we can isolate the reading from the server side scripts, and let a small high speed module do that for us.

That’s where Nginx and it’s Memcache Module comes in. We use PHP to fill up the cache and then let Nginx serve that content to the requesting client. According to lead architect Peter Gilg, this configuration is used at Mademan.com, a lifestyle site serving up to 1 million pageviews/day.

To be more specific, we’ll have nginx listen on port 80. If a request comes in, it checks memcache if the requested page is in memory and serves it directly to the client, or sends it to Apache (on port 88 for example) to produce the page. At the same time apache inserts the page into memcache, so it’s available to nginx for the next request. A sample configuration nginx script would look something like this:

server {
  location / {
    set $memcached_key $uri;
    memcached_pass     name:11211;
    default_type       text/html;
    error_page         404 = /fallback;
  }

  location = /fallback {
    proxy_pass backend;
  }
}

Additional Resources: Nginx Memcache Apache

Ever wondered it there is an easy way to resize your images or quickly create thumbnails from your favorite pictures?
Search no more, the awesome little utility ImageMagick does it for you.

Install it using your package manager, most of them should have it in their repository.

sudo apt-get install imagemagick

Once installed, this command will create tumbnails for all JPG’s in the current directory, 200 pixel wide/high on the longest side. Eg. if your image is in landscape layout, it’ll be 200 pixel wide, in portrait format it’ll be 200 pixel high.

for file in *.jpg ; do convert -resize 200 “$file” t”$file”; done

That’s it. Simply change 200 to the size of your liking, or change the extension if your images are in gif, png or any other image format.