Post mortem of the overnight DDoS Attack

Overnight on October 24, 2011 between the hours of approximately 12:55am and 2:55am ET the NS1 and NS2 anycast constellations came under a DDoS Attack. The attack was a combined syn flood and DNS flood which impaired the following anycast constellations to varying degrees:

ns1.easydns.com/remote1.easydns.com (a.k.a dns1.easydns.com)

ns2.easydns.com/remote2.easydns.com (a.k.a dns2.easydns.net)

 

The attack primarily impacted domains on the old platform or if domains on the new platform were affected, they were only delegated to the dns1 and dns2 nameserver constellations.

Scope of Impact

It is looking like the members affected were reporting problems primarily from the East and West Coasts of North America and seems to be (thankfully) limited, either by network geography and/or the time of day of the occurrence.

What You Can Do

If you are still on the old platform, please consider migrating to the new system sooner than later. We will actually be making an announcement about this fairly soon, but this is where everybody is headed anyway. On the new system at the very least, you get more nameservers: at least one more anycast strand and between 5 and 10 additional servers worldwide (as many as 16 more for Enteprise DNS customers). Our complete nameserver deployment on the new system is outlined here  – any impact reports last night from users on the new system were still using the old nameserver delegations from the old system or hadn’t added all available nameservers to their domains.

If you are on the new platform, please double check your nameserver delegations. If you are only delegated to DNS1 and DNS2, please add the appropriate additional nameservers.

• If we are your registrar, then you simply need to click on the “nameservers” link in the Domain Overview and select “Use easyDNS Nameservers”.
• If you are using an external registrar, reference this chart to see which nameservers to use for your delegation.

 

We We’re Doing

Target

We have located member domain which was the target of last night’s attack and upon examination found that it was violating the easyDNS AUP (they almost always are…) We have terminated service to this domain and it has since moved off of our nameservers.

Filters

Over the years we’ve found that the best way to mitgate a DOS attack is to not be the target of one, and we screen incoming domains with an eye toward filtering out “high probability” targets.  As we’ve remarked before:

Generally there are two kinds of DOS targets:

1. Targets you never knew were using your system and when you find out, you want to take a shower. These are the targets 90% of the time: ponzi schemes, virus distributors, phishing sites, etc. These are usually scumbags who make a lot of enemies and are already violating your AUP. The DOS brings your attention to their presence on your system, you throw them under a bus and the DOS follows them. Problem solved.
2. Then there’s “high profile” and “hot button” customers who people try to DOS just for bragging rights or some sort of vendetta. These are harder to handle, as you don’t want to cut off somebody who has a legit right to co-exist on the net with everybody else. Sometimes, you have to pull the plug, even temporarily to give yourself time to think and figure out your next move. I mean this in a very generic way: datacenters do this, network carriers do this, web hosts do this: if it gets too intense for your upstream, whammo, you’re null routed until things cool down. This is just how it works out here.

We consider last night’s target among the former. We have made additions to our signup filters that would block similar domains from coming here in the future.

Additional Nameserver Deployment

 

We are in the process of adding another node to the DNS1.easydns.com located at the BlackLotus datacenter in L.A. This will bring DNS1.easydns.com up to 6 nodes globally and adds some always welcome additional DOS-mitigation muscle to DNS1.

Migration

We are going to be accelerating our plans to migrate members away from the old platform and into the new one. The new easyDNS platform as more nameserver deployments and as we consolidate our member domains and have just one platform to deal with, we will have an easier time keeping it beefed up and expanding redundancy.

Conclusion

We are very sorry to any member who experienced issues because of this. We have striven over the years to be as redundant to DDoS attacks as possible, but as we have noted in our “DNS and DOS attacks” article (see below), even though anycast DNS deployments greatly decrease the scope of impact in a DDoS situation, it is still a major letdown for those people who are impacted. I want to personally extend apologies to those sysops out there were woken up at 1am on a Sunday night by their pagers and PDAs because their chosen nameserver vendor was in a firefight.

Further Reading:

• DNS and DOS Attacks: How to Stay UP When Your DNS Provider Goes dOWn