[Operators] Let's start some witch-hunt

Martin Sebald msebald at hot-chilli.net
Tue Jun 15 17:22:58 CDT 2010


Hello Peter, Mathias, Andreas and list,

great that Peter is bringing this issue up here.

Our server was dealing with two interesting things. One is what Peter
described here and the second is what Mathias described. I also think that
this is not a coincidence that our server flatlined several times since
this came up (almost at the same time, maybe three weeks ago).

I blacklisted new registrations for *lmc and *lop usernames for all our
domains and deleted all *lmc and *lop accounts. It got more quiet on the
server again, like before. But the log is full of this:

> I(<0.2524.11>:ejabberd_c2s:587) : ({socket_state,tls,{tlssock,#Port<0.514786>,#Port<0.514790>},<0.2523.11>}) Failed authentication for 8901260310018304070lop at jabber.hot-chilli.net
> I(<0.2532.11>:ejabberd_c2s:587) : ({socket_state,tls,{tlssock,#Port<0.514798>,#Port<0.514800>},<0.2531.11>}) Failed authentication for 1274536499966lop at jabber.hot-chilli.net
> I(<0.2518.11>:ejabberd_c2s:587) : ({socket_state,tls,{tlssock,#Port<0.514777>,#Port<0.514779>},<0.2517.11>}) Failed authentication for 89330119544679304170lop at jabber.hot-chilli.net
> I(<0.2543.11>:ejabberd_c2s:587) : ({socket_state,tls,{tlssock,#Port<0.514816>,#Port<0.514868>},<0.2542.11>}) Failed authentication for 1272130380669lop at jabber.hot-chilli.net

You can say for sure that at least one failed authentication message per
seconds comes in. So the old accounts are still trying to re-register plus
also new accounts try to register.

I also think more of a bot net or some kind or virus/worm/trojan (the idea
Andreas brought up). A new game which randomly creates accounts on several
server - I don't think so. Most for all because the game company /
developers might have asked if they can use the resources of our server.
This would be necessary to be sure that it does not happen that we close
doors and delete those accounts - and therefore limit the game itself.

Talking about the thing Mathias brought up: I did not do anything on our
server to stop this because I still think that it is the wrong thing to
limit j2j/xmpp transport to local users and/or blocking out Nimbuzz
completly. For sure this would work, but I think it is not the right way to
solve this problem.

Well, if the server gets DoS'd again, I will firewall Nimbuzz. I did not
find out yet what really caused this (logs look normal etc pp), so it is
hard to say.

The several DoS we experienced are pretty hard, see here:
http://jabber.hot-chilli.net/2010/05/31/server-flatlined-four-times-in-not-even-two-hours/

You have about 5-10 minutes to notice the problem and if you don't pkill
ejabberd, the server stops to respond to everything. Hard reset is the last
solution to get the machine up again.

Regards,
Martin



More information about the Operators mailing list