How To: Troubleshoot Exchange 2007 Queues Being Overrun With "Undeliverable" Messages

Like any good MSP… or maybe unlike many MSPs? ;-)… we have a monitor set placed on Exchange servers for when their queues get “out of hand,” i.e. there are too many outbound mail queues –or- the queues that are present are too large and have too many messages bound for a single destination.

Sometimes, when this alarm trips, it’s hard to know where to begin.  Did User McUserton decide to send out a blast e-mail to thousands of recipients… again?  Has the SMTP server been compromised via SMTP AUTH attack?  Is the outbound intermediary server down?  Has the server been blacklisted?  There are many scenarios here, of which this is just one; however, the troubleshooting steps here are a good starting point to see what you’re dealing with in most situations.

For this example, I was asked to investigate the presence of many outbound queues consisting of a few messages each—all of which are from a blank sender and have a subject line beginning with “Undeliverable”:

1) Hop on the Exchange server and open the Exchange Management Console

2) In the Microsoft Exchange tree, go to Toolbox and double-click on Queue Viewer:

   

3) Notice the characteristics of the scenario I outlined earlier (pictured below)… but even if your scenario doesn’t match this one, you can continue investigation… just pick a queue that looks the most “interesting” (e.g. has the most messages or seems the most backed up):

   

4) When you double-click on the queue, you’ll see a message or two… double-click on one of them to look at its properties:

   

    Things to notice here:

    Subject: Undeliverable: [Spammy subject line]

    From Address: <>

    Source IP: 255.255.255.255

    Last Error: [Delivery delay, DNS query failed, other failures]

5) In the example above, I’ve highlighted and copied “Watches, Luxury Items and Handbags!” to the clipboard, leaving out the “Undeliverable” bit… and now, back in the Exchange Management Console, go to the Toolbox > Message Tracking:

   

6) Un-check all of the checkboxes except for “Subject,” paste in the subject line you just copied, and hit Next:

   

7) Now… what you’ll see here is too much to put in a screenshot… here’s the trick… read it like a narrative, from left-to-right and top-to-bottom.  This takes practice and patience… hang in there, buddy!

   

    Here’s an example of what reading this narrative might sound like, with each line being a bullet point:

·         On June 2nd, 2011, a connection was logged from IP 8.9.10.11 and had a from address of <SpammyMcSpammerson@ILikeTacos.com> and a subject of “Watches, Luxury Items and Handbags!”, and this message was sent to <CEODistributionList@TheClient.com> and the result of this connection was “OK”

·         At the same date and a second later, [ExchangeServerName] tried to figure out who the <CEODistributionList@TheClient.com> distribution list should go to

·         At the same date and a second after that, the resolver figured out that the distribution list contains <BeckySue@TheClient.com> and <BillyBob@TheClient.com>

o    A second after that, the STOREDRV process attempts to deliver to <BeckySue@TheClient.com> and succeeds

o    At the same time, the STOREDRV process attempts to deliver to <BillyBob@TheClient.com>, but an error is logged saying that the recipient doesn’t exist

§  The [ExchangeServerName] then tries to send a message from <> (blank) to <SpammyMcSpammerson@ILikeTacos.com> with the subject “Undeliverable: Watches, Luxury Items and Handbags!”

§  A second after that, the transport driver says the recipient server said “4.4.0 – Unknown recipient” and rejected the message, so the transport driver put the queue into a retry state

§  Rinse, repeat, and presto change-o, you now have hundreds of queues stuck in a retry state!  Wooooo!

   

    Again, take your time and read slowly :-)

    Tip: You can select a message row and hit “Next” again to search for only that message ID.

          This will save you a lot of reading, especially if there are multiple messages with the same subject!

8) In my example above, we can see that the message originally hit the <CEODistributionList@TheClient.com> and then broke out to go to <BeckySue@TheClient.com> and <BillyBob@TheClient.com>, where the message to Billy Bob bounced (Billy Bob got a job down at Initech as one of the Bobs)… so two questions arise:

    Question 1: Why is Billy Bob still on the CEO’s distribution list?  We fired that guy!

    Solution 1: Remove Billy Bob from the CEO’s distribution list!  Disable his account!  Wipe hands on pants!

    Question 2: How in the heck did such an obvious piece of spam get through the spam filter?!?!

    Solution 2:  Check your spam filter!  In this case, we checked Postini… see if you can spot the problems:

   

Please use this knowledge for good and not for evil :-)

views
1 response
Awesome! the second part about using message tracking by subject was key to me finding that a old user that was disabled was still sending out an auto reply to all the spam she got. THANKS!!!