[TriLUG] Help diagnosing a hijacked mail server?

[Phillip Rhodes]

Hey guys, hoping someone can help me out a bit here.  I have a postfix
server running, serving one of my domains, and all of the "open relay
test" things tell me it's not relaying mail openly (and it shouldn't
be, I have postfix configured to require authentication to relay), but
somehow it's still sending spam like crazy.   I don't know if someone
brute-forced a password for one of the local accounts, or if the box
got rooted and something is running locally that is sending mail or
what.  And I'm not enough of a guru on the postfix logfile format to
really use the logs to interpret what's going on.

FWIW, here's a snippet of my /var/log/maillog:

Oct  9 08:40:52 fogbeam-int postfix/smtp[1734]: 578D84FE16:
to=<wgehlig at shaw.ca>, relay=idcmail.shaw.ca[24.71.223.11]:25,
delay=2314664, delays=2314601/63/0.14/0.18, dsn=2.0.0, status=sent
(250 ok:  Message 2081987273 accepted)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1759]: 4BBAB4D7B3:
to=<godworks2 at hotmail.com>, relay=mx1.hotmail.com[65.55.92.136]:25,
conn_use=9, delay=702, delays=639/63/0.04/0.21, dsn=2.0.0, status=sent
(250  <20121009082910.4BBAB4D7B3 at mail.fogbeam.com> Queued mail for
delivery)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1758]: 5EF114E000:
to=<arama.jones at aol.com>, relay=mailin-04.mx.aol.com[64.12.90.34]:25,
delay=2316992, delays=2316930/30/33/0, dsn=4.7.1, status=deferred
(host mailin-04.mx.aol.com[64.12.90.34] refused to talk to me: 421
4.7.1 : (DYN:T1) http://postmaster.info.aol.com/errors/421dynt1.html)
Oct  9 08:40:52 fogbeam-int postfix/error[1874]: 79EA9106D3F:
to=<braun.ankunft at kds-aktuell.de>, relay=none, delay=2180482,
delays=2180429/53/0/0.03, dsn=4.4.1, status=deferred (delivery
temporarily suspended: connect to mail.kds-aktuell.de[70.38.45.57]:25:
Connection timed out)
Oct  9 08:40:52 fogbeam-int postfix/error[1840]: 76F651FC52E:
to=<bastian-last at freier-widerstand.net>, relay=none, delay=2181495,
delays=2181442/53/0/0.03, dsn=4.4.3, status=deferred (delivery
temporarily suspended: Host or domain name not found. Name service
error for name=freier-widerstand.net type=AAAA: Host not found, try
again)
Oct  9 08:40:52 fogbeam-int postfix/qmgr[1590]: A8AB31FC636:
from=<godworks2 at hotmail.com>, size=1820, nrcpt=50 (queue active)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1747]: D5ED44D579:
to=<godworks2 at hotmail.com>, relay=mx1.hotmail.com[65.55.92.136]:25,
conn_use=15, delay=962, delays=898/63/0.04/0.24, dsn=2.0.0,
status=sent (250  <20121009082450.D5ED44D579 at mail.fogbeam.com> Queued
mail for delivery)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1611]: 578D84FE16:
to=<wgnradiosales at tribune.com>,
relay=chimx2.tribune.com[163.192.2.19]:25, delay=2314664,
delays=2314601/63/0.09/0.29, dsn=2.0.0, status=sent (250 ok:  Message
110817654 accepted)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1611]: 578D84FE16:
to=<wgntv-hr at tribune.com>, relay=chimx2.tribune.com[163.192.2.19]:25,
delay=2314664, delays=2314601/63/0.09/0.29, dsn=2.0.0, status=sent
(250 ok:  Message 110817654 accepted)
Oct  9 08:40:52 fogbeam-int postfix/qmgr[1590]: 4BBAB4D7B3: removed
Oct  9 08:40:52 fogbeam-int postfix/error[1848]: 79EA9106D3F:
to=<brandtwurm at stoertebeker.net>, relay=none, delay=2180482,
delays=2180429/53/0/0.08, dsn=4.4.1, status=deferred (delivery
temporarily suspended: connect to stoertebeker.net[82.98.86.162]:25:
Connection timed out)
Oct  9 08:40:52 fogbeam-int postfix/error[1900]: 94705106B94:
to=<becher.monster at tdg-germany.de>, relay=none, delay=2181367,
delays=2181313/52/0/1.9, dsn=4.0.0, status=deferred (delivery
temporarily suspended: host mx16.work.de[212.12.42.230] refused to
talk to me: 550-You are temporary blocked because you are listed on
www.nixspam.org. To 550 remove yourself from the list visit
http://www.dnsbl.manitu.net .)
Oct  9 08:40:52 fogbeam-int postfix/error[1798]: 527C81FD65B:
to=<dombau-oberdorf at stoertebeker.net>, relay=none, delay=2178518,
delays=2178465/53/0/0.87, dsn=4.4.1, status=deferred (delivery
temporarily suspended: connect to stoertebeker.net[82.98.86.162]:25:
Connection timed out)
Oct  9 08:40:52 fogbeam-int postfix/smtp[1698]: 078CF4DAEF:
to=<karenluu at fbi.gov>, relay=smtpc.fbi.gov[153.31.160.5]:25,
delay=625, delays=561/63/0.12/0.14, dsn=4.0.0, status=deferred (host
smtpc.fbi.gov[153.31.160.5] said: 452 Too many recipients received
this hour (in reply to RCPT TO command))


Given that, is there any way to tell where the smtp requests are
originating?  Any and all help is much appreciated.  Also, this really
needs to be fixed, and I'll gladly pay  a reasonable fee for someone's
expertise if this requires more in-depth work.


Thanks,


Phil