How to open .xz file in Linux/Unix?

.xz is a lossless file compression program that uses LZMA2 compression algorithm. xz is sort of stripped down version of 7-Zip program, although it has its own file format .7z

.xz compresses single file input and does not bundle multiple file in single archive. It is therefore a regular practice to use an archiving technique along with .xz compression. In Linux/Unix it is popularly, tar.

How to open/decompress/extract .tar.xz file?

$ tar -xJf yourfile.tar.xz

-J is the flag to handle .xz files in tar program.

Although, modern tar program can automatically determine what compression technique is used and apply thereby. You can simply run the following and it should work

$ tar -xf yourfile.tar.xz

Converting mbox to maildir

mbox and maildir both are popularly used techniques to store mails. The simple way to understand the difference is, mbox uses a single directory to store all mails while maildir would utilize a directory structure to store mails. Most of the cases, advance mail receiving techniques IMAP over POP3 uses these sort of folder based concept to provide enhancement in mail organizing.

If you are having a POP3 account using mbox, you can use an open source tool called mb2md to convert it to maildir.

Full reference and how to use the tool is available here:
http://batleth.sapienti-sat.org/projects/mb2md/

How to convert mbox to maildir using mb2md?

First download the mb2md script from the developer

$ wget http://batleth.sapienti-sat.org/projects/mb2md/mb2md-3.20.pl.gz
$ gzip -d mb2md-3.20.pl.gz
$ chmod 755 mb2md-3.20.pl

Now, converting is simple. Use as following:

$ mb2md-3.20.pl -s /locationofmbox -R -d /destinationofmaildir

-s defines the source of the mbox, the location would depend on the type of environment you are using.
-R defines recursively do for all subdirectories
-d defines the destination of maildirectory inbox.

How to convert mbox to maildir in a directadmin instance manually?

For Example, if you are using Directadmin control panel, the mbox spool is located under /var/spool/virtual/domain.com/user. You would want to run something like this to convert the mbox into maildir in a directadmin control panel:

$ mb2md-3.20.pl -s /var/spool/virtual/domain.com/user -R -d /home/username/imap/domain.com/user/Maildir
$ chown -Rf username:mail /home/username/imap/domain.com/user/Maildir

Install rtorrent in RHEL/CentOS

The easiest way to install rtorrent (a command line bittorrent client) is to use the RPMForge Repository. You can install RPMForge using the following artcle How to install RPMForge Repository in RHEL/Centos.

Once you are done, you just need to run the following command to install rtorrent in your RHEL/CentOS (5/6):

yum install libtorrent rtorrent libsigc

libsigc++ is a dependency library for libtorrent. Yum should include it automatically in the installation. RPMForge has the compatible libsigc++ in their repository.

rtorrent: symbol lookup error: rtorrent: undefined symbol

How to fix the error “rtorrent: symbol lookup error: rtorrent: undefined symbol”?

If you have installed rtorrent using RPMForge Repository, you are possibly seeing the error with libtorrent “0.13.2-1”. There is a conflict between the latest libtorrent and rtorrent. The easiest way, I have found to solve the error is to downgrade the libtorrent version using the same repository. Run the following to downgrade the libtorrent:

yum downgrade libtorrent

This should downgrade your libtorrent version to 0.12.9-2, which seems to work fine with rtorrent 0.8.9-2.

How to install RPMForge Repository in RHEL/Centos

RPMforge is a collaboration of Dag and other packagers. They provide over 5000 packages for CentOS, including wine, vlc, mplayer, xmms-mp3, and other popular media tools. It is not part of Red Hat or CentOS but is designed to work with those distributions.

How to install RPMForge in RHEL 5/Centos 5?

Run the following command to download and install RPMForge in RHEL 5 based systems:

i386:

rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.i386.rpm

x86_64:

rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm

How to install RPMForge in RHEL 6/Centos 6?

Run the following command to download and install RPMForge in RHEL 6 based systems:

i386 (32bit):

rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.i686.rpm

x86_64 (64bit):

rpm -ivh http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm

You can install DAG’s GPG key as following:

rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt

Now you can install packages from RPMForge, like

yum install libtorrent

To view the complete list of RPMForge Packages, visit the following page:

http://packages.sw.be/

How to open .gz file in Linux/Unix?

You need to use gzip/gunzip program to open/extract a .gz file. It is based on DEFLATE algorithm. All the Linux/Unix comes with default gzip/gunzip program as it is used as primary compression technique in Linux/Unix based system.

How to extract a .gz file?

gunzip yourcompressfile.gz

or

gzip -d yourcompressfile.gz

-d refers to decompress the gzip file.

To view the latest file, type

ls -l

How to extract tar.gz file?

tar.gz files are tar files compressed with gunzip. You can use the tar program in Linux to extract a tar.gz file as following:

tar -xvzf yourfile.tar.gz

Here, x = extract, v = verbose, z = gzip, f = force

If the tar file isn’t compressed with gunzip, then you need to exclude the ‘z’ option which would be as following:

tar -xvf yourfile.tar

A Brilliant App Optimization/Monitoring Tool – New Relic!

Almost 24 hours ago, one of my friend referred to me an interesting offer from ‘tutplus’

http://dev.tutsplus.com/articles/get-a-free-year-of-tuts-premium-by-trying-new-relic–cms-12

It seems Tutplus either affiliated or owned a new App optimization tool named “New Relic”. My primary objective was of course to get the free Tut+ Premium for a year and the Nerd T-shirt, and whats hard in deploying a PHP App Monitoring tool in one of the server! So I started.

The deployment of the tools are fairly easy. I am not really in the Mobile App thing, so I had chosen the PHP Web App monitoring tool. The deployment is well instructed. Its a RPM based installer for RHEL based releases, pretty clean and simple. Once the installation was done, it added a shared object in my PHP interpreter and started grabbing data. Out of a surprise, I started seeing details that are really cool. Things like “Errors” and “Stack Trace” are the finest invention of this tool. The Stack trace gives you reports like “strace” which is my favorite tool of linux debugging facility. The basic advantage of this feature in New Relic is, it saves the data and post you as a token in the dashboard of new relic. Now, isn’t it brilliant? I sorted almost 23 major bugs in client’s account since I have installed the monitor. Database monitoring also includes some exceptional features that are not usually available in App Monitoring/Optimizations tools I had used before.

Unfortunately, the tool is free for 2 weeks. Since then, the “Pro” version comes with 150$ a month per host. The price is certainly high, but the result is truly amazing, looking at the features and performance of the tool.

At the end of all, I had my Tut+ premium for one year for free of charge and a nerd T-shirt on the way to my home 😀

If you haven’t tried it, you can try it now. If you are an android developer, you can add the code in your app, and monitor your App for 14 days for free, and get a Tut+ premium for free for a year.

Just for a record, I am not affiliated with neither Tut+ nor New Relic. The link should not contain any affiliate url.

Happy troubleshooting!

How to track all outgoing mails in Exim

If you are a mail server administrator and possibly using one of the most used open source mail server namely Exim, you might require to monitor the outgoing mails to track down a spammer. In shared web servers, you can use some regular expressions on the mail logs to trace a spammer log. But sometimes, you might fail to find a possible spammer if you have a huge amount of users in the server and a lot of users are actually spamming. Most of the cases, user’s accounts are compromised and intruders utilize the facility to send out spam.

Sometimes, a better way is to store a copy of the each mails sent using Exim and use the regular expressions on the header details to track down the original spammer. Just for the record, storing email data may breach the privacy concern if it is a public server and this should only be used to track the original spammer.

Exim can utilize 3 levels of mail filtering. I have used System Filtering to deliver a copy of each mail sent to a local mailbox. A system filter works for all the accounts and users under Exim. In Cpanel, you can set the exim filter from Cpanel >> Service Configuration >> Exim Configuration Manager >> Filter

From command line, open the /etc/exim.conf and find the line starts with “system_filter”.

By default, cpanel uses a system filter located under “/etc/cpanel_exim_system_filter”. Just for the record, this copy will always get reverted to the default on each cpanel update. We need to make a customized filter for our use. I did the following:

cp /etc/cpanel_exim_system_filter /etc/exim_system_filter_mellowhost

I made a copy of the original system filter to exim_system_filter_mellowhost. Now open the copy with your favorite text editor, mine is always nano.

Now, you need to add some simple shell script inside this custom filter using Exim filtering commands you can find here:

http://www.exim.org/exim-html-3.30/doc/html/filter_29.html

Here is a shortcode I have used:

if first_delivery
and ("$h_from:" does not contain "[email protected]")
and not ("$h_X-Spam-Checker-Version:" begins "SpamAssassin")
then
unseen deliver "[email protected]"
endif

Just for the record, “localdelivery.com” is an account I have created under the same server. I don’t own the domain neither operate it. I have used it to create a local inbox and deliver the mails for me. You just need to make sure the domain lies in /etc/localdomains. That is the local resolver for Exim and it won’t go for a dns resolution check if the domain is available under localdomains which serves our purpose. You need to make sure, you create an email account with the localdelivery.com, in my case, I created an individual inbox with “[email protected]”.

Now, here is the breakdown of the shortcode. “first_delivery” means the mail is just sent, it hasn’t been queued or relayed. “$h_from” is a variable used by exim to determine the from address from each mail header. So, I am checking whether the mail is the just dispatched from a mail user and whether the mail was ever delivered to [email protected] or not using the 2nd line where it checks if the from “does not contain” (an exim filter command) our local delivery mail address. If the line isn’t included, your mail forwarder will fall in an infinite loop and keep forwarding your own mails to yourself.

The third condition is included if you have SpamAssassin installed to check your mails for spam. SpamAssassin is an individual daemon that will check every first delivery of mails, add its spam score in the header and send the mail again. That would make Exim realize the mail is another “First Delivery”. So, if the SpamAssassin score is added, we are safely discarding them as we have already received those mails in our local inbox.

Now the production of the all clauses is very simple. It is delivering the mail as “unseen deliver” (an exim command, means make the mail unread in the inbox” to our localdelivery inbox.

How can you trace down the spammer from an aggregated inbox?

It depends on how would you like to use regular expressions and tools like “grep, awk, cut” etc. Let me give you some insight on basic.

First of all, all these mails are actually getting stored as text files under the local mail directory. In my case, it is under “/home/localdel/mail/localdelivery.com/tracker/”.

Now move your shell prompt to the folder “cur” (current mails in mail directory). If you check the files, you should see each mails are stored as one individual text file.

In my case, I usually sort the subjects first and track down if there is spammer out there. You can do that using the following:

grep -i "Subject: " *

This would result all the subject and the file name.

One of my favorite way to track down a spammer is to check for Duplicate subjects. You can do it as following:

cat *|grep "Subject: "|cut -d":" -f2|sort|uniq -c|sort -n

cut is a tool to divide the sentence using regular expression and print the part you want. In my case, I am dividing the Subject lines with “:” and printing the 2nd column which is our original subject. Now we are sorting the result alphabetically with “sort”. Counting the unique values with “uniq -c” and sorting them again from low to high using sort -n.

This was just the basic of using parsing and trace out as spammer. The more you work with the spam mails, the more you will understand. Parsing talent learns based on experience.

Happy Troubleshooting!

Why are we using Softlayer Nameservers?

I was reviewing the live chat transcripts earlier today. An interesting one that was served by “Ronskit”, a live chat operator of Mellowhost caught my attention. One of our visitor was interested to know, why are we using Softlayer nameservers for the domain “mellowhost.com” (http://intodns.com/mellowhost.com) instead of ns1.mellowhost.com or so on. The visitor was more interested in proving that Mellowhost is hosted in a shared server and all of our clients are also using a server that is not really managed by Mellowhost. His excuses were flowing towards why we don’t sell VPS or Master Resellers, or so called “Alpha” Master Resellers instead we only sell Reseller and Shared Hosting. It is eventually hard to answer a management level of query by a sales representative and as expected he wasn’t able to please the visitor 🙂 I quickly thought to write this down for future references.

 

Continue reading “Why are we using Softlayer Nameservers?”

Experience with Varnish!

When Mellowhost first launched her servers, all of them were using “DSO” module for serving php. I can remember, one of the most commonly used caching plugin was either Eaccelerator or Xcache. Eaccelerator was preferable as cpanel Easyapache have this in their option and can be compiled automatically while rebuilding apache. As time passed, we had to choose suphp instead of dso due to many factors involved. I hope to write them down at later time why we had to move to suphp. But that actually cut the idea of using dynamic cacher in the server. Suphp kills the php process after serving, this allows all the opcode cacher to be valueless. Due to cutting off a dynamic cacher, the server started showing pretty good load average. Although it was a good trade off of IO and CPU usage in cost of security. I was searching for a cacher that would work with suphp in the same technique Litespeed (A paid web server software) does.

Continue reading “Experience with Varnish!”