Talk about the latest detailed news on botnets! The ultimate of an inside look at botnets – it is the real, first-hand account of what happened this week when a zombie botnet woke up, based on some unseen signal triggered or programmed by the botnet owner, and took over hundreds of customer computers at a large US broadband ISP.
This particular large U.S. broadband ISP had been preparing for – and expecting – this day for nearly two and a half years. The day when their customer’s computers that had been compromised and coopted as part of a zombie botnet, would be called to wake up and start doing their dirty work – in this case spewing out millions of pieces of spam from their customers’ computers. They knew it had to come. And come it did, and it kept on coming.
Here is how it went down, in the words of one of the ISP’s own security staff:
“Around 16:15h EDT, a group of about 350 customer machines woke
up in a time frame of about 2 minutes, and started spewing (email)
through our outbound, relaying MTAs at a rather high rate.
We detected and mitigated about 245 of these via automated
means in the first 4 minutes, and the activity subsided a
good 10 minutes later. No final count on the amount of email
enqueued vs. dropped by content filters, yet, but I am estimating
that it was less than 120,000 mails escaping to world, total.”
Not so bad, really. Readily contained, it had seemed.
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
Except that was only the first wave. Perhaps the test wave. Because less than 4 hours later, it started up again, and this time it was rocking and rolling.
“At that time we started detecting and mitigating
machines at over 300 customers per minute, but the detection
rate exceeded the mitigation facility’s maximum speed significantly,
creating a backlog as bad as 40 minutes from detection to
mitigation (which will be addressed), and amplified by later
(re-)detections that signalled that mail activity from a customer IP
didn’t stop within a minute of being (supposedly) stopped.”
In other words, this ISP’s broadband network was being flooded with outgoing email (spam) being sent by hundreds of computers connected to their network. Computers belonging to their own customers. And why were these customers’ computers doing this?
Because these customers had allowed their computers to becoming infected and, unknowingly, part of this zombie botnet! (Read about the Secure Your Computer and Take Back the Net initiative here.
Number of customer machines that had their SMTP service shut off, per minute:
Between 6:05pm and 6:10pm – in only 10 minutes they had to cut off nearly 1000 customer computers that had become part of this botnet.
Before all was said and done, more than 2250 of their customers computers had woken up and started sending spam, under the command of the botnet.
Here are the number of machines waking up and spewing spam over the course of a botnet attack that lasted more than one-and-half hours, in a minute-by-minute report, as reported first hand, starting at 6:02pm and ending at 7:59pm
“1 2007/05/29 18:02:
7 2007/05/29 18:04:
32 2007/05/29 18:05:
81 2007/05/29 18:06:
138 2007/05/29 18:07:
360 2007/05/29 18:08:
305 2007/05/29 18:09:
48 2007/05/29 18:10:
62 2007/05/29 18:11:
62 2007/05/29 18:12:
35 2007/05/29 18:13:
34 2007/05/29 18:14:
39 2007/05/29 18:15:
32 2007/05/29 18:16:
46 2007/05/29 18:17:
88 2007/05/29 18:18:
34 2007/05/29 18:19:
19 2007/05/29 18:20:
28 2007/05/29 18:21:
39 2007/05/29 18:22:
49 2007/05/29 18:23:
60 2007/05/29 18:24:
67 2007/05/29 18:25:
53 2007/05/29 18:26:
58 2007/05/29 18:27:
38 2007/05/29 18:28:
19 2007/05/29 18:29:
13 2007/05/29 18:30:
17 2007/05/29 18:31:
11 2007/05/29 18:32:
17 2007/05/29 18:33:
44 2007/05/29 18:34:
33 2007/05/29 18:35:
22 2007/05/29 18:36:
14 2007/05/29 18:37:
8 2007/05/29 18:38:
24 2007/05/29 18:39:
12 2007/05/29 18:40:
13 2007/05/29 18:41:
7 2007/05/29 18:42:
9 2007/05/29 18:43:
11 2007/05/29 18:44:
9 2007/05/29 18:45:
10 2007/05/29 18:46:
9 2007/05/29 18:47:
6 2007/05/29 18:48:
7 2007/05/29 18:49:
3 2007/05/29 18:50:
11 2007/05/29 18:51:
9 2007/05/29 18:52:
3 2007/05/29 18:53:
3 2007/05/29 18:54:
6 2007/05/29 18:55:
4 2007/05/29 18:56:
5 2007/05/29 18:57:
1 2007/05/29 18:58:
2 2007/05/29 18:59:
6 2007/05/29 19:00:
7 2007/05/29 19:01:
3 2007/05/29 19:02:
6 2007/05/29 19:03:
7 2007/05/29 19:04:
1 2007/05/29 19:05:
4 2007/05/29 19:06:
7 2007/05/29 19:07:
9 2007/05/29 19:08:
3 2007/05/29 19:09:
5 2007/05/29 19:10:
2 2007/05/29 19:11:
2 2007/05/29 19:12:
3 2007/05/29 19:13:
1 2007/05/29 19:14:
8 2007/05/29 19:15:
1 2007/05/29 19:16:
1 2007/05/29 19:17:
3 2007/05/29 19:18:
4 2007/05/29 19:20:
6 2007/05/29 19:21:
4 2007/05/29 19:22:
2 2007/05/29 19:24:
2 2007/05/29 19:25:
4 2007/05/29 19:26:
1 2007/05/29 19:27:
1 2007/05/29 19:28:
2 2007/05/29 19:29:
2 2007/05/29 19:30:
1 2007/05/29 19:31:
2 2007/05/29 19:32:
2 2007/05/29 19:33:
3 2007/05/29 19:34:
1 2007/05/29 19:35:
1 2007/05/29 19:36:
4 2007/05/29 19:38:
7 2007/05/29 19:39:
2 2007/05/29 19:40:
2 2007/05/29 19:41:
4 2007/05/29 19:42:
4 2007/05/29 19:44:
8 2007/05/29 19:45:
1 2007/05/29 19:48:
2 2007/05/29 19:49:
4 2007/05/29 19:50:
2 2007/05/29 19:52:
3 2007/05/29 19:53:
3 2007/05/29 19:54:
3 2007/05/29 19:55:
4 2007/05/29 19:56:
3 2007/05/29 19:57:
5 2007/05/29 19:58:
3 2007/05/29 19:59:”
At the end of the day, here is what the attack – which turned out to be Russian spam – looked like:
” – Total zombie customers detected and SMTP-filtered: over 3300
– Total number of spams making it out: in excess of 1,023,000
– Total number of spams getting dropped by outbound content
filtering (which worked well for the first 15 minutes, until the
spammers apparently adapted the content): over 257,000
– Queue size: 500K at max, still at 25K after 18 hrs.”
So, for any of you working at ISPs, this is what you have to look forward to.
And for all of you who are customers of ISPs, for pete’s sake, secure your computer, now!
The Internet Patrol is completely free, and reader-supported. Your tips via CashApp, Venmo, or Paypal are appreciated! Receipts will come from ISIPP.
CashApp us
Venmo us
Paypal us
if any of the ip’s took the time to blacklist certain things, this would have never happened. about the stupidest thing i’ve ever seen anyone do with a botnet.
Huey – you’re absolutely right.
Gary – if you do some investigating – e.g. search nanas for reports on that date, between 18:00 and 20:00, you’d probably be able to ID them by identifying a spike.
Place the responsibily with those that allowed the breach so the informed consumer could react accordingly.
The ‘responsibily’ lies with the thousands of end-users who managed to get themselves trojanned, but anybody with experience at an ISP knows that that’s something that exists anywhere. What makes this ISP different is that 1) they were already forcing their customers to go through their smarthosts, 2) they already had an automatic abuse detection and mitigation system, and 3) they had live staff that could react to the problem in real-time.
The name of the ISP who reported this would be a PR coup for them, because they’re doing significantly better than most of the rest of the industry.
If the name of the ISP was also reported this would be a newsworthy story.
Inagine an analogous storyline “A lage bank in the US had all their credit card numbers and autorization codes stolen” without naming the bank.
Place the responsibily with those that allowed the breach so the informed consumer could react accordingly.