The Real Profile of a Zombie Botnet Waking Up and Taking Over an ISP’s Customers Computers   - 2,393 Views, 4 Comments

This particular large U.S. broadband ISP had been preparing for - and expecting - this day for nearly two and a half years. The day when their customer’s computers that had been compromised and coopted as part of a zombie botnet, would be called to wake up and start doing their dirty work - in this case spewing out millions of pieces of spam from their customers’ computers. They knew it had to come. And come it did, and it kept on coming.

Here is how it went down, in the words of one of the ISP’s own security staff:

“Around 16:15h EDT, a group of about 350 customer machines woke
up in a time frame of about 2 minutes, and started spewing (email)
through our outbound, relaying MTAs at a rather high rate.
We detected and mitigated about 245 of these via automated
means in the first 4 minutes, and the activity subsided a
good 10 minutes later. No final count on the amount of email
enqueued vs. dropped by content filters, yet, but I am estimating
that it was less than 120,000 mails escaping to world, total.”

Not so bad, really. Readily contained, it had seemed.

Except that was only the first wave. Perhaps the test wave. Because less than 4 hours later, it started up again, and this time it was rocking and rolling.

“At that time we started detecting and mitigating
machines at over 300 customers per minute, but the detection
rate exceeded the mitigation facility’s maximum speed significantly,
creating a backlog as bad as 40 minutes from detection to
mitigation (which will be addressed), and amplified by later
(re-)detections that signalled that mail activity from a customer IP
didn’t stop within a minute of being (supposedly) stopped.”

In other words, this ISP’s broadband network was being flooded with outgoing email (spam) being sent by hundreds of computers connected to their network. Computers belonging to their own customers. And why were these customers’ computers doing this?

Because these customers had allowed their computers to becoming infected and, unknowingly, part of this zombie botnet! (Read about the Secure Your Computer and Take Back the Net initiative here.

Number of customer machines that had their SMTP service shut off, per minute:

Between 6:05pm and 6:10pm - in only 10 minutes they had to cut off nearly 1000 customer computers that had become part of this botnet.

Before all was said and done, more than 2250 of their customers computers had woken up and started sending spam, under the command of the botnet.

Here are the number of machines waking up and spewing spam over the course of a botnet attack that lasted more than one-and-half hours, in a minute-by-minute report, as reported first hand, starting at 6:02pm and ending at 7:59pm

“1 2007/05/29 18:02:
7 2007/05/29 18:04:
32 2007/05/29 18:05:
81 2007/05/29 18:06:
138 2007/05/29 18:07:
360 2007/05/29 18:08:
305 2007/05/29 18:09:
48 2007/05/29 18:10:
62 2007/05/29 18:11:
62 2007/05/29 18:12:
35 2007/05/29 18:13:
34 2007/05/29 18:14:
39 2007/05/29 18:15:
32 2007/05/29 18:16:
46 2007/05/29 18:17:
88 2007/05/29 18:18:
34 2007/05/29 18:19:
19 2007/05/29 18:20:
28 2007/05/29 18:21:
39 2007/05/29 18:22:
49 2007/05/29 18:23:
60 2007/05/29 18:24:
67 2007/05/29 18:25:
53 2007/05/29 18:26:
58 2007/05/29 18:27:
38 2007/05/29 18:28:
19 2007/05/29 18:29:
13 2007/05/29 18:30:
17 2007/05/29 18:31:
11 2007/05/29 18:32:
17 2007/05/29 18:33:
44 2007/05/29 18:34:
33 2007/05/29 18:35:
22 2007/05/29 18:36:
14 2007/05/29 18:37:
8 2007/05/29 18:38:
24 2007/05/29 18:39:
12 2007/05/29 18:40:
13 2007/05/29 18:41:
7 2007/05/29 18:42:
9 2007/05/29 18:43:
11 2007/05/29 18:44:
9 2007/05/29 18:45:
10 2007/05/29 18:46:
9 2007/05/29 18:47:
6 2007/05/29 18:48:
7 2007/05/29 18:49:
3 2007/05/29 18:50:
11 2007/05/29 18:51:
9 2007/05/29 18:52:
3 2007/05/29 18:53:
3 2007/05/29 18:54:
6 2007/05/29 18:55:
4 2007/05/29 18:56:
5 2007/05/29 18:57:
1 2007/05/29 18:58:
2 2007/05/29 18:59:
6 2007/05/29 19:00:
7 2007/05/29 19:01:
3 2007/05/29 19:02:
6 2007/05/29 19:03:
7 2007/05/29 19:04:
1 2007/05/29 19:05:
4 2007/05/29 19:06:
7 2007/05/29 19:07:
9 2007/05/29 19:08:
3 2007/05/29 19:09:
5 2007/05/29 19:10:
2 2007/05/29 19:11:
2 2007/05/29 19:12:
3 2007/05/29 19:13:
1 2007/05/29 19:14:
8 2007/05/29 19:15:
1 2007/05/29 19:16:
1 2007/05/29 19:17:
3 2007/05/29 19:18:
4 2007/05/29 19:20:
6 2007/05/29 19:21:
4 2007/05/29 19:22:
2 2007/05/29 19:24:
2 2007/05/29 19:25:
4 2007/05/29 19:26:
1 2007/05/29 19:27:
1 2007/05/29 19:28:
2 2007/05/29 19:29:
2 2007/05/29 19:30:
1 2007/05/29 19:31:
2 2007/05/29 19:32:
2 2007/05/29 19:33:
3 2007/05/29 19:34:
1 2007/05/29 19:35:
1 2007/05/29 19:36:
4 2007/05/29 19:38:
7 2007/05/29 19:39:
2 2007/05/29 19:40:
2 2007/05/29 19:41:
4 2007/05/29 19:42:
4 2007/05/29 19:44:
8 2007/05/29 19:45:
1 2007/05/29 19:48:
2 2007/05/29 19:49:
4 2007/05/29 19:50:
2 2007/05/29 19:52:
3 2007/05/29 19:53:
3 2007/05/29 19:54:
3 2007/05/29 19:55:
4 2007/05/29 19:56:
3 2007/05/29 19:57:
5 2007/05/29 19:58:
3 2007/05/29 19:59:”

At the end of the day, here is what the attack - which turned out to be Russian spam - looked like:

” - Total zombie customers detected and SMTP-filtered: over 3300

- Total number of spams making it out: in excess of 1,023,000

- Total number of spams getting dropped by outbound content
filtering (which worked well for the first 15 minutes, until the
spammers apparently adapted the content): over 257,000

- Queue size: 500K at max, still at 25K after 18 hrs.”

So, for any of you working at ISPs, this is what you have to look forward to.

And for all of you who are customers of ISPs, for pete’s sake, secure your computer, now!