How To Configure mdadm RAID e-mail Notifications via ISP SMTP Using msmtp

[fermulator]

This tutorial will assist a home server administrator to configure mdadm e-mail notifications without a full fledged MTA (such as postfix or exim).

Estimated Time: 1 hour

My Goals:
* If a drive fails in any of my RAID arrays, I'd like to be notified by e-mail. The server should tell me, I shouldn't have to go looking for failures.
* E-mail configuring should be as simple as possible, and should authenticate against my ISP's SMTP server. I don't want to have to configure a full Mail Transfer Agent (MTA).

Disclaimer:
* I take no responsibility if, after following these guidelines, your server fails to notify you by e-mail of a failed drive. This tutorial is accurate to the best of my knowledge, and has been tested to work, but use at your own risk. If you have important or critical data, and you're not completely comfortable with linux server administration, please, get a professional to setup your RAID.

Caveats:
* I'm obviously running a particular setup and distro, but hopefully one could take this instructions and apply them to their own configuration as needed.

System Specs:
* Ubuntu 10.04.4 LTS Server (2.6.32-44-generic-pae)
* mdadm v2.6.7.1
* msmtp 1.4.19
* 3x RAID Arrays : /dev/md250 (3x250GB), /dev/md500 (5x500GB), /dev/md1000 (4x1TB)

Preconditions & Assumptions:
I assume here that you have a working server, with a working RAID device already configured. This tutorial does not cover how to setup or configure RAID, but instead how to configure e-mail notifications for an already working mdadm RAID array.

Background Information:
* Firstly. How does e-mail work? View the operations and diagram from wikipedia here.
* Secondly, here's what our pieces of the puzzle are for e-mail:

MUA (the client): msmtp (a sendmail compatible interface)
MTA (the mail server): your isp or gmail
MSA (smtp middle man): msmtp (a simple MTA that gets mail from your local MTA to your real MTA or mailhub)

* As you can see, to simplify things, we'll be using msmtp as both the MUA and the MSA.
* mdadm e-mail notification by default looks for /usr/sbin/sendmail.

Process the First - Installing and Configuring the MSA (msmtp)

So firstly, we need to install msmtp. On ubuntu, this is trivial:

sudo apt-get install msmtp msmtp-mta

Now it's time to configure. There are two methods for msmtp configuration; system wide, and user base. We're interested in a system wide configuration which will apply to all users, unless otherwise overrided by a user's own msmtp configuration file.

At the time of writing, the version of msmtp I was using denoted that the system-wide configuration file was /etc/msmtprc. It is recommended for you to confirm the same via msmtp --version:

$ msmtp --version | grep "System configuration"
System configuration file name: /etc/msmtprc

Here is a sample system configuration file:

sudo nano /etc/msmtprc

# ------------------------------------------------------------------------------
# msmtp System Wide Configuration file
# ------------------------------------------------------------------------------

# A system wide configuration is optional.
# If it exists, it usually defines a default account.
# This allows msmtp to be used like /usr/sbin/sendmail.

# ------------------------------------------------------------------------------
# Accounts
# ------------------------------------------------------------------------------

# Sympatico Account
account sympatico
host smtphm.sympatico.ca
from noreply@yourserver.name
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
user valid_email@bell.net
password ********
syslog LOG_MAIL

# Rogers Account
account rogers
host smtp.broadband.rogers.com
port 587
from noreply@yourserver.name
auth login
user valid_email@rogers.com
password ********
syslog LOG_MAIL

# gmail account
# Configuring for gmail is beyond the scope of this tutorial
# Googling for "gmail msmtp" should help
account gmail
host smtp.gmail.com
port 587
from noreply@yourserver.name
tls on
tls_starttls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
auth on
user <username>
password ********
syslog LOG_MAIL


# Other ISP Account
# Configuring for other ISPs is beyond the scope of this tutorial
# Googling for "myisp outlook smtp" should help

# ------------------------------------------------------------------------------
# Configurations
# ------------------------------------------------------------------------------

# Construct envelope-from addresses of the form "user@oursite.example".
#auto_from on
#maildomain fermmy.server

# Use TLS.
#tls on
#tls_trust_file /etc/ssl/certs/ca-certificates.crt

# Syslog logging with facility LOG_MAIL instead of the default LOG_USER.
# Must be done within "account" sub-section above
#syslog LOG_MAIL

# Set a default account
account default : rogers

# ------------------------------------------------------------------------------

Save and close this file.

See http://msmtp.sourceforge.net/doc/msm...guration-files for additional details.

Now it's time to test that e-mail works! First, we'll run a "pretend test" to make sure the settings we've just configured have taken affect.

msmtp --pretend

Verify the output is correct as best as you know it to be.

Now, we'll try an actual test.

echo "This is a test e-mail from my server using msmtp" | msmtp -d youremail@domain.com

If all went well, you should receive an e-mail very shortly!

If things didn't go well, you can check the mail log files (/var/log/mail.log) for hints as to what went wrong, or perhaps even the msmtp executable will spit out some useful errors.

Step the Second - Installing and Configuring our MUA ("sendmail")

Now, as per the homepage of msmtp, msmtp is a "Sendmail compatible interface (command line options and exit codes)."

This is exciting because it means that if msmtp is compatible with sendmail commands and exit codes, we can simply use msmtp as our e-mail command line executable.

*NEW*

• Feedback from other users, and I personally tested on Ubuntu 10.04 LTS, has yielded that installing msmtp-mta performs the proper sendmail linkage automatically for us. Since this was part of the installation instructions above, we can proceed without this next code block.
• Should something go wrong with mdadm sending e-mail via "sendmail", please consider manually over-riding the sendmail symlink to msmtp as follows:

sudo mv /usr/sbin/sendmail /usr/sbin/sendmail.bak
sudo ln -s /usr/bin/msmtp /usr/sbin/sendmail

*FEEDBACK REQUESTED*

• If you performed this how to, and did not require the manual sendmail symlink, please reply to this thread confirming. Thanks.

Step the Third - Configuring MDADM for e-mail notifications

Edit the mdadm configuration file:

sudo nano /etc/mdadm/mdadm.conf

NOTE: Most distributions expect the mdadm.conf file in /etc/, not /etc/mdadm. I believe this is a "ubuntu-ism" to have it as /etc/mdadm/mdadm.conf.

I'll avoid details about the DEVICE declarations and such that might already exist in this configuration file, and instead we'll focus on the MAIL options. If you're interested in more details on the mdadm.conf, check the man page here: http://man-wiki.net/index.php/5:mdadm.conf.

# instruct the monitoring daemon where to send mail alerts
MAILADDR youremail@domain.com
MAILFROM my-server-name - mdadm

Save and close this file.

That's it!

Let's run a simple test from mdadm to ensure e-mail notifications are working. If you have DEVICE declarations in your mdadm.conf file, use the following command:

sudo mdadm --monitor --scan --test --oneshot

This should deliver an e-mail, scanning all devices found in mdadm.conf.

If you rely on mdadm scanning your partitions for RAID devices, you'll want to specify the devices (usually found as /dev/md###):

sudo mdadm --monitor --scan --test --oneshot /dev/md[[:digit:]]*

This should deliver an e-mail, with the results from /proc/mdstat. The "/dev/md[[:digit:]]*" portion of the command should match all RAID devices starting with "md" for you automatically.

I then received the following e-mail (actually x3, one for each of my arrays because I have 3):

-------- Original Message --------
From: my-server-name - mdadm
Sent: Sat 18 May 2013 09:39:59 PM EDT
To: myemail@domain.com
Cc: Subject: TestMessage event on /dev/md640:my-server-name

This is an automatically generated mail message from mdadm
running on fermmy-server

A TestMessage event had been detected on md device /dev/md640.

Faithfully yours, etc.

P.S. The /proc/mdstat file currently contains the following:

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md2000 : active raid6 sdl1[1] sdn1[3] sdk1[0] sdm1[2] sdj1[5] sdi1[4]
7814053632 blocks super 1.1 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

md1001 : active raid1 sda1[1] sdb1[0]
9764800 blocks [2/2] [UU]

md250 : active raid1 sdb3[0] sda3[1]
233392960 blocks [2/2] [UU]

md1002 : active raid1 sda2[1] sdb2[0]
1952704 blocks [2/2] [UU]

md1000 : active raid5 sdd1[0] sde1[1] sdc1[3] sdf1[2]
2930279808 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

md640 : active raid1 sdh1[0] sdg1[1]
625130708 blocks super 1.1 [2/2] [UU]

unused devices: <none>

Step the Fourth - Configuring MDADM for Monitoring

At this stage of the game, you're (hopefully) almost done! The only part left is to ensure that your linux distribution is already monitoring your RAID array, and if it's not, configure it yourself.

In short, a "monitor" mdadm process must always be running which periodically checks the status of your array(s). This is typical of any RAID system. It's my understanding that the mdadm monitor will send out e-mail alerts on the following events:

Fail, FailSpare, DegradedArray, and TestMessage

These events really are the only things we would typically care to receive an e-mail about. Good.

Now, a typical mdadm monitoring mode process command might like like:

mdadm --monitor --scan --daemonize --test --syslog (/dev/md[[:digit:]]*)

Let's break down the above command:
* --monitor --> Sets the mdadm process to "Monitor Mode".
* --scan --> Scans the mdadm.conf file for RAID DEVICES.
* --daemonize --> This process should be treated as a daemon.
* --test --> On system startup, send out test e-mail(s).
* --syslog --> Log to the system log (/var/logs/syslog)
* The last portion is optional(?) for those who don't have devices defined in the RAID configuration file.

For addition details on mdadm monitor mode, try mdadm --monitor --help or review the man file for monitor mode at http://man-wiki.net/index.php/8:mdadm#For_Monitor_mode:.

For my case in particular, Ubuntu 10.04, the monitoring of all RAID arrays had already been implemented, and started on system startup. This was realized by the file /etc/init.d/mdadm, and it's debian configuration options defined in /etc/default/mdadm.

I prefer to have a test e-mail on startup, so in the /etc/default/mdadm option file, I changed

DAEMON_OPTIONS="--syslog"

to

DAEMON_OPTIONS="--syslog --test"

or (OPTIONAL)

DAEMON_OPTIONS="--syslog --test (/dev/md[[:digit:]]*)"

NOTE: As per the beginning of "Step the Fourth", if you don't have any devices defined in your mdadm.conf configuration file, you need to suffix the DAEMON_OPTIONS with "(/dev/md[[:digit:]]*)"

To verify that the mdadm monitor process is actually running, try:

ps aux | grep mdadm

Result in my case, showing the monitor process is already running:

root 6230 0.0 0.0 2160 600 ? Ss May29 0:04 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog --test

or (OPTIONAL)

root 21021 0.0 0.0 2160 584 ? Ss 20:12 0:00 /sbin/mdadm --monitor --pid-file /var/run/mdadm/monitor.pid --daemonise --scan --syslog --test /dev/md1000 /dev/md250 /dev/md500

If your linux distribution doesn't already have the mdadm monitoring daemon, you'll have to configure this yourself, but I'm afraid this is beyond the scope of this tutorial. I suggest Googling "mdadm monitor daemon" for help, and in particular, this is useful: http://linux-raid.osdl.org/index.php...ng_RAID_arrays.

Good luck! Once you know a mdadm monitor daemon is running, you can simulate a failure to verify that the system will indeed e-mail you, which is what we're after right?
* I found this website very useful: http://linux-raid.osdl.org/index.php..._drive_failure

By this point, you should find comfort in the knowledge that your server will tell YOU when a drive fails!

Credits:

This tutorial was created from me conducting research, trial and error, and from some specific content from a few different pages, so I want to give credit where credit is due!
* http://www.novell.com/support/viewCo...1034&sliceId=1
* http://ubuntuforums.org/showthread.p...ighlight=msmtp

[iBurger]

Thank you! This is exactly what I was looking for.!!

[tjhart85]

Thanks for this! This is exactly what I needed!

[danger89]

Everything works OK, also send e-mail using msmtp until...


root@server:~# sudo mdadm --monitor --scan --test --oneshot
sendmail: fatal: open /etc/postfix/main.cf: No such file or directory

I will get that error message, I think the symlink link didn't work, what to do?

EDIT:
I just installed postfix again, could be via:
sudo apt-get install --reinstall postfix

Or first --remove & --purge, then --install

[fermulator]

Everything works OK, also send e-mail using msmtp until...


root@server:~# sudo mdadm --monitor --scan --test --oneshot
sendmail: fatal: open /etc/postfix/main.cf: No such file or directory

I will get that error message, I think the symlink link didn't work, what to do?

EDIT:
I just installed postfix again, could be via:
sudo apt-get install --reinstall postfix

Or first --remove & --purge, then --install

Did that resolve your issue?

[lunix-]

Thanks a lot for sharing your knowledge!

This was a fun howto to complete and it worked like a breeze on my Debian Lenny server.

This is great stuff for us semi-newbies!!

[braincookie]

Wow, that was a nice one, it worked like a charm on ubuntu Jaunty. I want to point out one minor mistake you made: you estimated it would take one hour to complete this, but with those great explanations, it took me less than 15 minutes

Thanks a lot!

[Evil Dax]

Nice, clean how-to, exacly what I need.
However, somebody's not happy:

dax@fileserver:/$ echo "This is a test." | msmtp -d my@isp

--
*bunch of stuff*
--


<-- 501 5.6.0 Data format error: we do not accept messages without any headers
msmtp: the server did not accept the mail
msmtp: server message: 501 5.6.0 Data format error: we do not accept messages without any headers
msmtp: could not send mail (account default from /etc/msmtprc)

So, can I add a bunch of headers somewhere?

[Carlbc18]

Works great, exactly what i needed.

[joachim.buechse]

I think the "kosher" way to get the /usr/sbin/sendmail link is to install the package msmtp-mta (which depends on msmtp).

Step the Second - Installing and Configuring our MUA ("sendmail")

Now, as per the homepage of msmtp, msmtp is a "Sendmail compatible interface (command line options and exit codes)."

This is exciting because it means that if msmtp is compatible with sendmail commands and exit codes, we can simply use msmtp as our e-mail command line executable.

Create a symlink for sendmail to msmtp:

sudo ln -s /usr/bin/msmtp /usr/sbin/sendmail

Caveat: I'm not sure if this is "kosher" in general with the Linux community. This method worked for me, but if someone has a "proper" solution, I'd be more than happy to hear it.