Ffmpeg is the de-facto open source standard for encoding audio and video. Especially with the arrival of html5 which uses codecs like Ogg, Theora, Vorbis, h264, webm among others, and the increasing popularity of devices from Apple, such as iPod, iPad and the iPhone, all of which require a different standard from flv, the version so widely in use by Adobe Flash.

Another positive aspect is that ffmpeg has a much revived developer group, posed to enhance the application on a much faster pace than it has been in recent years.

Installation on Linux systems is really straight forward, especially on common distro’s like Ubunty and Fedora. On a Mac, we will have to build it from source (if you’re not a fan of ports like me), including encoder/decoders we plan to use. Here is the breakdown to get ffmpeg installed on a Mac running Snow Leopard.

Install Git

If you don’t have Git installed, download the Git installer here, install the package and run the script in the source of the mounted image. CVS is included in XCode and that is located on the snow leopard installation disc.

Install LAME (latest version from cvs)

cd ~
cvs -d:pserver:anonymous@lame.cvs.sourceforge.net:/cvsroot/lame login
(at the password prompt, press Enter)
cvs -z3 -d:pserver:anonymous@lame.cvs.sourceforge.net:/cvsroot/lame co -P lame
cd lame
./configure
make
sudo make install

Install FAAC and FAAD

cd ~
wget http://sourceforge.net/projects/faac/files/faac-src/faac-1.28/ \
  faac-1.28.tar.gz/download
tar xzvf faac-1.28.tar.gz
cd faac-1.28
./configure --with-libmp4v2
sudo make install

cd ~
wget http://sourceforge.net/projects/faac/files/faad2-src/faad2-2.7/ \
  faad2-2.7.tar.gz/download
tar xzvf faad2-2.7.tar.gz
cd faad2-2.7
./configure
make
sudo make install

Install Yasm (for faster encoding)

cd ~
wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
tar xzvf yasm-1.1.0.tar.gz
cd yasm-1.1.0
./configure
make
sudo make install

Install Ogg, Vorbis and Theora

cd ~
wget http://downloads.xiph.org/releases/ogg/libogg-1.1.4.tar.gz
tar xzvf libogg-1.1.4.tar.gz
cd libogg-1.1.4
./configure
make
sudo make install

cd ~
wget http://downloads.xiph.org/releases/vorbis/libvorbis-1.2.3.tar.gz
tar xzvf libvorbis-1.2.3.tar.gz
cd libvorbis-1.2.3
./configure
make
sudo make install

cd ~
wget http://downloads.xiph.org/releases/theora/libtheora-1.1.1.tar.bz2
tar xzvf libtheora-1.1.1.tar.bz2
cd libtheora-1.1.1
./configure
make
sudo make install

Install ffmpeg

cd ~
svn checkout svn://svn.ffmpeg.org/ffmpeg/trunk ffmpeg
cd ffmpeg
./configure --enable-gpl --enable-version3 --enable-nonfree --enable-postproc \ 
--enable-pthreads --enable-libfaac --enable-libfaad2 --enable-libmp3lame \ 
--enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora \
--enable-libx264 --enable-libxvid --enable-x11grab --enable-libvpx-vp8

Note

As this article matures so will new versions of the above extensions become available. Especially webm will undoubtedly see a rapid development during the coming months and you should ensure that you use the latest versions of each.

Sharing files between VM’s can be a pain, especially if your VM is running on a text based server without UI. For example, I start up a new VM for almost anything and I deal with up to 20 different sites and application and every one has it’s own VM.

VMWare provides it’s “Guest Additions” and that includes folder sharing. It works fine on desktop os’s, but on the command line we have to mount it using VMWare’s file system, which can be a bit cumbersome. A much simpler and easier way to share files is via USB or DropBox. Getting the DropBox sync daemon to run is really easy and once running, anything you add to DropBox will be synced to your VM transparently and at network speed.

If you don’t already have a DropBox account setup, get a free one here. Now on to the steps to get the sync daemon up and running.

Steps:

SSH to your Linux server and change to your home directory.
cd ~
for 32-bit os:
wget -O dropbox.tar.gz “http://www.dropbox.com/download/?plat=lnx.x86″
for 64-bit os:
wget -O dropbox.tar.gz “http://www.dropbox.com/download/?plat=lnx.x86_64″
After the file “dropbox.tar.gz” downloaded properly, then extract it
tar -xvzf dropbox.tar.gz
Run dropboxd
~/.dropbox-dist/dropboxd
The following error message is expected: This client is not linked to any account…
Please visit https://www.dropbox.com/cli_link?host_id=123456789qwertyuiop to link this machine.

Copy / paste this url into the browser of your desktop machine and log in. This is the process of where your VM will be authorized and added to the account. Once completed, the daemon dropboxd should have created a ~/Dropbox folder and synchronize your files.

Note:

If you want to change the account it is linked to, unlink it from the first account, then kill the running dropbox process, start it up again (with “~/.dropbox-dist/dropboxd &”) and obtain the new host_id with dbreadconfig.py . If you don’t restart the dropbox client, it will give the same host_id (which for some reason cause me to be unable to change the account it is linked to).

Mac OS 10.6.5 + apachectl: /usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument

After the upgrade to Mac OS 10.6.5 a bug seem to have snuck in where apachectl tries to re-set ulimit to unlimited. The error appears as such:

/usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument

While apachectl was working flawless before the upgrade to Mac OS 10.6.5, I suspected the two were related. After a quick search into what the update entailed, I found that Apple upgraded to Apache 2.2.15, which in turn patched a number of vulnerabilities. As it turned out, one of those changes broke the apachectl script.

Referring to line 82 of apachectl, the shell script was referring to the ULIMIT_MAX_FILES variable that was trying to set to unlimited, which read:

ULIMIT_MAX_FILES="ulimit -S -n `ulimit -H -n`"

Changing the line as follows fixed the problem:

ULIMIT_MAX_FILES="ulimit -S -n 1024"

Should you be running a high amount of vhosts or do some testing that requires a significant amount of file descriptors, set this number higher accordingly.

Growing companies at some point are faced with the increased workload to setup, configure and maintain their servers and applications. Administrators try to script repetitive tasks and ultimately look for solutions that are scalable and provide a central management tool base. There are several software packages available, commercial and open source ones.

Puppet is probably the most used open source configuration management package available today and more system administrators and software developers are interested in setting it up within their environment. An excellent book about Puppet (Pulling Strings with Puppet) is available and so are numerous online articles and tutorials. However, lots of online resources are just not quite complete and can lead to lots of frustration, especially in the setup and configuration of Puppet.

So here is another step by step tutorial about how to install and setup Puppet to the point where we can run a simple recipe residing on the server (puppetmaster) and execute on the client. For this illustration we’ll install Puppet on Ubuntu 10.10 using a deb package.

Configure Network and DNS

Setting up configure hostnames on server and client is crucial. Since Puppet uses self signed certificates to secure communication, it’s essential that network is setup properly and hostnames are configured with fully qualified domain names.

Let’s open /etc/hosts and add the necessary entries. (note: If you use a DNS server in your network, those setting changes should be made on your DNS server and not the hosts file)

server hosts file (assuming server = 192.168.1.10, client = 192.168.1.20):

127.0.0.1 localhost.localdomain localhost puppet
192.168.1.10 puppet-server01.example.com puppet-server01 puppet
192.168.1.20 puppet-client01.example.com puppet-client01

verify:

# hostname => puppet-server01
# hostname -f => puppet-server01.example.com

client hosts file:

127.0.0.1 puppet-client01.example.com localhost
192.168.1.10 puppet-server01.example.com puppet-server01 puppet

verify:

# hostname => puppet-client01
# hostname -f => puppet-client01.example.com

In addition, you should ensure that no firewall is blocking port 8140, as Puppet is communicating on this port.

Now we’re ready to move on to installing the puppet server.

Before installing and configuring puppet server (puppetmaster), ensure that your network is properly setup. Configure Puppet Server on Ubuntu 10.10 is easy and straight forward. It’s those subleties about the network configuration and necessary puppet recipes and files that can make our lives miserable.

To install Puppet, choose one of the following methods (we will use the Ubuntu Package)

Using package manager of your distro

• RPM (Fedora) for Red Hat, Fedora (puppet-server for server, puppet for client)
• DEB (Debian) for Ubuntu, Debian (puppetmaster for server, puppet for client)

Installing from source: (note: you will need to have Ruby installed on your system prior to installing Puppet)

• download the latest version from puppetlabs
• # tar -zxf puppet-0.xx.x.tgz
• # cd puppet-0.xx.x
• # ruby install.rb
• # puppet –version (to verify installation)

Using Ruby Gem: (note: you will need to have Ruby and Ruby Gem installed on your system prior to installing Puppet)

# gem install –remote –source http://reductivelabs.com/downloads puppet

Create a puppet user and group, in case your intallation method did not automatically create them for you:

• # groupadd puppet
• # useradd –M –g puppet puppet

Create a manifest. Without manifest (a text document that defines a particular configuration), puppetmaster will trow an error:

• # mkdir –p /etc/puppet/manifests
• # mkdir –p /etc/puppet/files
• # vi /etc/puppet/manifests/site.pp (and add the following puppet class)

package {
‘htop’ :
ensure => installed
}

Start the server (we start it manually to ensure it works without errors, later we will start it as a daemon)

# puppetmasterd –verbose –no-daemonize

We should get confirmation that the server has started and displays the current version. Any error messages should get resolved at this point before moving on to Install and configure puppet client on Ubuntu 10.10.

Using diff to create batch files and subsequently running them against select files is such a convenient way to update, fix or change existing files, but many developers and system administrators simply don’t know much about them.

Here is a quick primer on using diff and apply changes for select files. Use diff –help to check out more options and flags.

NOTE: pay caution when using patching, any mistake or error are executed without warning or undo feature. It’s best to always make a backup of any file or folder that are going to be affected by your patch.

diff

Some useful flags you can specify when comparing files are -b (ignore white space difference), -B (ignore blanc lines), -r (recursive) and -i (ignore case).

Let’s say we need to change the GA analytics code on our pages and that they are unfortunately hard coded. The files are shown below as are the results from diff:

File GA_v1:

var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-9876543-21']);
_gaq.push(['_setDomainName', '.domain.com']);
_gaq.push(['_trackPageview']);

(function() {
var ga = document.createElement(‘script’); ga.type = ‘text/javascript’; ga.async = true;
ga.src = (‘https:’ == document.location.protocol ? ‘https://ssl ‘ : ‘http://www’) + ‘.google-analytics.com/ga.js’;
var s = document.getElementsByTagName(‘script’)[0]; s.parentNode.insertBefore(ga, s);
})();

File GA_v2:

var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-1234567-89']);
_gaq.push(['_trackPageview']);

(function() {
var ga = document.createElement(‘script’); ga.type = ‘text/javascript’; ga.async = true;
ga.src = (‘https:’ == document.location.protocol ? ‘https://ssl’ : ‘http://www’) + ‘.google-analytics.com/ga.js’;
var s = document.getElementsByTagName(‘script’)[0]; s.parentNode.insertBefore(ga, s);
})();

File GAold.patch, resulting from running diff -bi GA_v1 GA_v2 > GA_v2.patch:

2c2,3
<   _gaq.push(['_setAccount', 'UA-1234567-89']);
—
>   _gaq.push(['_setAccount', 'UA-9876543-21']);
>   _gaq.push(['_setDomainName', '.domain.com']);
7c8
<     ga.src = (‘https:’ == document.location.protocol ? ‘https://ssl’ : ‘http://www’) + ‘.google-analytics.com/ga.js’;
—
>     ga.src = (‘https:’ == document.location.protocol ? ‘https://ssl <https://ssl/> ‘ : ‘http://www’) + ‘.google-analytics.com/ga.js’;

“2c2,3″ = copy line 2 (from GA_v1) to line 2 (of GA_v2), insert line 3
“7c8″ = copy line 7 to line 8

(c – copy, d – delete, a – add)

patch

And with this file available, we now can use patch to perform the changes. One really important flag is --dry-run It does not actually change any files, just print what would happen. And -b will make a backup of your files. Please use it especially if you’re new to patching.

patch –dry-run GA_v1 < GA_v2.patch

If you get no errors, you’re good to go.

patch -b GA_v1 < GA_v2.patch (using -b to make a backup)

Doing a diff of GA_v1 and GA_v2 shows a difference of 1 empty line. We could have avoided that using the -B flag when creating the patch file using diff, but there are many more options for both, diff and patch commands and we encourage you to explore them.

Ever wanted to listen, track and parse tweets from a Twitter stream from an individual user? It’s quite an easy task, if you know the right URL’s to parse. Curl is a great little tool to get sources from almost any web resource and Curl has roots in Linux command like and PHP, just to name a couple.

Let’s look at how this would work by pulling CNN’s breaking news feed as JSON:

curl http://twitter.com/status/user_timeline/cnnbrk.json

or XML:

curl http://twitter.com/status/user_timeline/cnnbrk.xml

That will give you the last 20 tweets in either JSON or XML format.

An even more interesting option is to get messages where you have been mentioned. It’s a bit more complicated, as we need to supply login credentials, but no rocket science either:

curl -u “username:password” http://www.twitter.com/statuses/mentions.json (or .xml if you prefer)

That’ll give you the last few tweets mentioning the user supplied with the curl command. Now all you got to figure out is how to parse the messages and plug them into your application.

What’s the proper method to run a script at startup on Ubuntu?
You can use update-rc.d for start-only or stop-only scripts, following these steps:

Start script called “startup_script” on startup (note the dot at the end of the line) :

# update-rc.d -f startup_script start 99 2 3 4 5 .

- start is the argument given to the command (start, stop).
- 99 is the start priority of the script (1 = first one, 99= last one)
- 2 3 4 5 are the runlevels at which to run the script

The dot at the end of the line has significance, read more here: /etc/rcS.d/README

Start startup_script on shutdown and reboot :
# update-rc.d -f startup_script start 90 0 6 .

Stop startup_script on halt and reboot :
# update-rc.d -f startup_script reboot 90 0 6 .

To run the script as a daemon, use the skeleton file located at “/etc/init.d/skeleton”

To know which runlevel you are running, type:
$ runlevel

Read more about runlevels here.

Memcache is traditionally used as a module inside server side scripts, such as PHP, ASP, ColdFusion and others. And it’s doing a terrific job, as long as it’s implemented correctly.

But if we look under the hood of the actual Memcache application, and I’m not talking about the PHP or ASP extension, but rather the executable that’s running as a daemon under linux, for example, it is a rather simple database like application running in memory. Now there are two basic actions that need to be performed to use it, one is writing info into memcache, the other is reading it. In a typical scenario, there are many reads for one write, that’s the whole point, isn’t it. But what if we can isolate the reading from the server side scripts, and let a small high speed module do that for us.

That’s where Nginx and it’s Memcache Module comes in. We use PHP to fill up the cache and then let Nginx serve that content to the requesting client. According to lead architect Peter Gilg, this configuration is used at Mademan.com, a lifestyle site serving up to 1 million pageviews/day.

To be more specific, we’ll have nginx listen on port 80. If a request comes in, it checks memcache if the requested page is in memory and serves it directly to the client, or sends it to Apache (on port 88 for example) to produce the page. At the same time apache inserts the page into memcache, so it’s available to nginx for the next request. A sample configuration nginx script would look something like this:

server {
  location / {
    set $memcached_key $uri;
    memcached_pass     name:11211;
    default_type       text/html;
    error_page         404 = /fallback;
  }

  location = /fallback {
    proxy_pass backend;
  }
}

Additional Resources: Nginx Memcache Apache

Ever wondered it there is an easy way to resize your images or quickly create thumbnails from your favorite pictures?
Search no more, the awesome little utility ImageMagick does it for you.

Install it using your package manager, most of them should have it in their repository.

sudo apt-get install imagemagick

Once installed, this command will create tumbnails for all JPG’s in the current directory, 200 pixel wide/high on the longest side. Eg. if your image is in landscape layout, it’ll be 200 pixel wide, in portrait format it’ll be 200 pixel high.

for file in *.jpg ; do convert -resize 200 “$file” t”$file”; done

That’s it. Simply change 200 to the size of your liking, or change the extension if your images are in gif, png or any other image format.