5 Replies Latest reply on Jan 18, 2008 11:04 PM by 807574

    Error read/writng SMTP packet

    807574
      Sun Java System Messaging Server 6.2-3.04

      For a few days, I've had messages to remote domains queued up with one of two errors: "Error writing SMTP" packet or "Error reading SMTP packet; response to dot-stuffed message expected; likely problem with network or remote SMTP server." Most of our mail is being delivered to most domains, but some mail to some domains is being re-queued over and over. For instance, one message to domain.edu gets through fine; another is queued for days; a third was queued with one of the errors for an hour or two, and then is delivered. There's no pattern to the messages (size, attachments, senders, recipients).

      master_debug on TCP_local shows (with a bit of snipping):
      2:04:37.71: Sending : "MAIL FROM:<aaa@oursite.com> SIZE=4382720
      12:04:37.84: Got status : "250 OK <aaa@oursite.com> Sender ok"
      12:04:37.84: Sending : "RCPT TO:<bbb@theirsite.com>
      12:04:37.88: Got status : "250 OK <bbb@theirsite.com> Recipient ok"
      12:04:37.88: Sending : "DATA"
      12:04:37.92: Got status : "354 Start mail input; end with <CRLF>.<CRLF>"
      12:04:37.92: Write message header/body in one go
      12:06:49.18: smtp_pmt_write: [0x00000024] network write failed
      12:06:49.22: smtp_pmt_close: [0x00000024] status 0

      ********************************
      Second type of log:
      2:00:24.13: Sending : "MAIL FROM:<aaa@oursite.com> SIZE=25600
      12:00:25.13: Got status : "250 2.1.0 <aaa@oursite.com>... Sender ok"
      12:00:25.13: Sending : "RCPT TO:<bbb@theirsite.com>
      12:00:26.52: Got status : "250 2.1.5 bbb@theirsite.com... Recipient ok"
      12:00:26.52: Sending : "DATA"
      12:00:26.79: Got status : "354 Start mail input; end with <CRLF>.<CRLF>"
      12:00:26.79: Write message header/body in one go
      12:00:26.80: ... Message header/body, 370 lines ...
      12:00:26.80: Sending : "."
      12:03:27.16: smtp_pmt_read: [0x00000024] network read failed
      12:03:27.17: smtp_pmt_close: [0x00000024] status 0

      From what I've read here and elsewhere, these failure indicate a network issue; we aren't receiving the expected ack, we time out, and re-queue.

      Since I'm seeing this happening with both of our outbound servers, and multiple receiving domains, is it reasonable to start looking at our network and/or firewall? What else should I look at? Nothing is being dropped at the firewall, per our firewall guy.

      Thanks for any help.
        • 1. Re: Error read/writng SMTP packet
          807574
          LesliStClair wrote:
          Sun Java System Messaging Server 6.2-3.04
          Although this isn't necessarily related to your issue you may want to consider updating to a more recent patch release e.g. 118207-63 (the last public 6.2 patch).
          For a few days, I've had messages to remote domains queued up with one of two errors: "Error writing SMTP" packet or "Error reading SMTP packet; response to dot-stuffed message expected; likely problem with network or remote SMTP server."
          If you look through your mail logs, can you point to a particular time when these errors seemed to increase substantially?

          If so this may act as a reference point to see when/if there have been changes in your environment (messaging server, network etc.) which could explain the behaviour change.
          Most of our mail is being delivered to most domains, but some mail to some domains is being re-queued over and over. For instance, one message to domain.edu gets through fine; another is queued for days; a third was queued with one of the errors for an hour or two, and then is delivered.
          For messages that are successfully delivered after initially failing, is there a pattern to the times when these succeed (e.g. redelivery at night during 'quiet' times)?
          There's no pattern to the messages (size, attachments, senders, recipients).
          ********************************
          Second type of log:
          2:00:24.13: Sending : "MAIL FROM:<aaa@oursite.com> SIZE=25600
          12:00:25.13: Got status : "250 2.1.0 <aaa@oursite.com>... Sender ok"
          12:00:25.13: Sending : "RCPT TO:<bbb@theirsite.com>
          12:00:26.52: Got status : "250 2.1.5 bbb@theirsite.com... Recipient ok"
          12:00:26.52: Sending : "DATA"
          12:00:26.79: Got status : "354 Start mail input; end with <CRLF>.<CRLF>"
          12:00:26.79: Write message header/body in one go
          12:00:26.80: ... Message header/body, 370 lines ...
          12:00:26.80: Sending : "."
          -> 3 minutes to get a response. This can indicate either the destination server took too long to process the contents of the message (didn't return 250 ok after we finished sending the email) or that the destination server never got the entire mail in time and hence didn't return a 250 ok.
          12:03:27.16: smtp_pmt_read: [0x00000024] network read failed
          12:03:27.17: smtp_pmt_close: [0x00000024] status 0

          From what I've read here and elsewhere, these failure indicate a network issue; we aren't receiving the expected ack, we time out, and re-queue.
          Correct.
          Since I'm seeing this happening with both of our outbound servers, and multiple receiving domains, is it reasonable to start looking at our network and/or firewall?
          Yes that is a reasonable assumption to make.
          What else should I look at? Nothing is being dropped at the firewall, per our firewall guy.
          What you can try is to single out a problem email that has a consistent destination server IP address, then run a network snoop to capture the traffic to this server. Then try to 'return' that message manually and review the snoop using a tool like wireshark.

          Regards,

          Shane.
          • 2. Re: Error read/writng SMTP packet
            807574
            Thanks, Shane. We're replacing the servers in a couple of months, so there's not much point in patching now.

            As for environment changes--there haven't been any on the email end of things, but I can't speak for the network or firewall. I've only tracked one message that was queued, and then delivered, but it wasn't delivered at a particularly quiet time (mid-afternoon) nor was it sent at a particularly busy time. If that were the issue, I'd expect the backed up mail to go out overnight, but instead, I have some things queued up until they expire.

            Looks like I'll be recruiting the network staff to trace the transaction. Thanks.
            • 3. Re: Error read/writng SMTP packet
              807574
              Resolution: firewall issue. Nokia's Smart Defense rules sporadically dropped good traffic due to a buffer issue.
              • 4. Re: Error read/writng SMTP packet
                807574
                It's amazing how often we find firewalls to be at fault, when we get strange SMTP issues. It turns out that firewall companies aren't very good at building SMTP systems, and the best thing to do is to turn off all filtering, testing, etc, and have the firewall simply forward the port directly to the Messaging Server. There are no known security exploits for any current version of Messaging Server's MTA.
                • 5. Re: Error read/writng SMTP packet
                  807574
                  Right. The SmartDefense thing was supposed to monitor only, and apparently was supposed to protect us from rogue Exchange servers or something. Not quite! Hope my experience can at least save someone else a headache or two.