Menu Home

Uncluttering Postfix queues: bulk removal of Undelivered notifications

After deploying check_mk monitoring across a client’s server fleet, it became clear that our previous monitoring wasn’t sufficiently watching postfix queues. Fortunately the impact was minor. We had hosts with sometimes just a couple of dozen non-critical queued mails, we did have one with several tens of thousands of queued mails from a faulty cronjob though.

One of the exacerbating issues is the number of chatter messages, usually undelivered mail/return to sender messages.

You know I’ve figured out some complicated way to deal with it, right? So let’s start:

idArray=( $(mailq | awk '{print $1}' | egrep -v '@|\(|\-|root' | grep . | while read -r messageid; do 
  printf "%s\n" "${messageid} $(postcat -q "${messageid/\*/}" | egrep 'Subject|Date')" 
done | grep -B 1 Undelivered | egrep -v '\--|Subject' | awk '{print $1}') )

So what we’re doing here is getting a list of object ID’s from the queue, then for each of them we’re printing out the ID and the object’s details. From that we look for items with “Undelivered” somewhere around or in the Subject line. For any that are found, we add them to an array, the aptly named “idArray

for id in ${idArray[@]}; do postsuper -d $id; done

Short and sweet. Anything in idArray gets removed from the mail queue. There are probably neater ways to do this like:

printf "%s\n" "${idArray[@]}" | postsuper -d -

But the for loop is easy, familiar and does the job.

mailq | awk '{print $1}' | egrep -v '@|\(|\-|root' | grep . | while read -r messageid; do
  printf "%s\n" "${messageid} $(postcat -q "${messageid}" | egrep 'Subject|Date')"
done

Now we can inspect what’s left and see if there’s something obvious that we can fix. Much like this, sanitised real world example:

$ mailq | awk '{print $1}' | egrep -v '@|\(|\-|root' | grep . | while read -r messageid; do
>   printf "%s\n" "${messageid} $(postcat -q "${messageid}" | egrep 'Subject|Date')"
> done
13C81112 Subject: Anacron job 'cron.daily' on blarhost.tld
Date: Sat,  2 Apr 2016 03:38:12 +1300 (NZDT)
CAF20115 Subject: Anacron job 'cron.daily' on blarhost.tld
Date: Thu, 31 Mar 2016 03:10:43 +1300 (NZDT)
563F8113 Subject: Anacron job 'cron.daily' on blarhost.tld
Date: Mon,  4 Apr 2016 03:10:13 +1200 (NZST)
6D94A111 Subject: Anacron job 'cron.daily' on blarhost.tld
Date: Sun,  3 Apr 2016 03:40:20 +1200 (NZST)
9A4BD11E Subject: Anacron job 'cron.daily' on blarhost.tld
Date: Fri,  1 Apr 2016 03:20:12 +1300 (NZDT)
$
$
$ postcat -q 9A4BD11E
*** ENVELOPE RECORDS deferred/9/9A4BD11E ***
message_size:             611             202               1               0             611
message_arrival_time: Fri Apr  1 03:20:12 2016
create_time: Fri Apr  1 03:20:12 2016
named_attribute: rewrite_context=local
sender_fullname: Anacron
sender: root@blar.tld
original_recipient: root
recipient: root@blar.tld
*** MESSAGE CONTENTS deferred/9/9A4BD11E ***
Received: by blarhost.tld (Postfix, from userid 0)
        id 9A4BD11E; Fri,  1 Apr 2016 03:20:12 +1300 (NZDT)
From: Anacron 
To: root@tld
Content-Type: text/plain; charset="ANSI_X3.4-1968"
Subject: Anacron job 'cron.daily' on blarhost.tld
Message-Id: <20160331142012.9A4BD11E@blarhost.tld>
Date: Fri,  1 Apr 2016 03:20:12 +1300 (NZDT)

/etc/cron.daily/logrotate:

error: syslog.orig:1 duplicate log entry for /var/log/cron
error: found error in /var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
, skipping
*** HEADER EXTRACTED deferred/9/9A4BD11E ***
*** MESSAGE FILE END deferred/9/9A4BD11E ***

Categories: Geeking Out Lunix Lunacy Tech Tips

rawiri