Post-mortem of the June 3-4th DDoS

As most of you are aware, we were DDoS-ed yesterday and experienced severe impact across our dns1, dns2 and dns3 anycast clouds.

There was no identifiable target here on the system, rather, it appears as if our infrastructure was used in an attempt to DDoS somebody else, in other words, it was a DNS amplification attack. While most of these typically use open resolvers, it is also now common to  use authoritative nameservers in reflection attacks.

In our case while the attack didn’t take up too much bandwidth, the number of inbound packets soon filled up our connection tables and due to the way the attack was constructed, it was difficult to discern real DNS queries from fake ones. With a little help from our mitigation partners we muddled through from about 5pm June 3rd until midnight or 2am June 4th, and then thanks to our colleagues at DNSMadeEasy and DNSimple (the latter of which, we believe were victim the same attack the day before yesterday), we got a handle on things around 5:30am EST this morning.

What We’re Doing Now

• We are reconfiguring all of our nameservers to handle these types of attacks better. Many of these filters are in place now and showing good results.
• We’re provisioning additional mitigation gear via Staminus (dns1 and dns3)

What We’re Going To Do Next

• We’re initiating a shift in the way we route traffic through Prolexic (dns2) and upgrading our hardware there
• We’re going to make it easier to turn up warm spare nameservers as discussed in our workaround post and our original “Surviving a DDoS agianst your DNS provider” post.
• We’re smoothing out the algorithms for Proactive Nameservers – which some people turned on last night, and got good initial feedback for a new service like this. But in some cases the algo would flap and it would switch nameservers pools in and out too frequently.

Key Takeaways

For us:

This is hard to admit, but we got careless. While we correctly believed Sunday’s “mini-DDoS” was a test run, and we thought we were ready for the anticipated “real deal”, some of our gear wasn’t configured optimally.  We should have better insight into DNS attack patterns and we didn’t recognize this as an amplification attack until quite late into the game.

 

For you:

We’ve said it before, we’re going to say it again, if you absolutely, positively need to have 100% DNS availability all the time, then the way to do it is to use multiple DNS solutions, whether that’s multiple commercial providers or your own DNS servers in your own datacenters or elsewhere. It’s best to have all this setup in advance, as we lay out in the Surviving DoS attacks. (We added our Route53 nameservers right away and thus the easyDNS control panel, etc was up throughout it all.)

• We can recommend DNSMadeEasy, DNSimple or No-Ip, then there’s Route53 (our users had good results with easyRoute53 overnight).
• You can export your bind zonefiles here, and there is even an API call to automate that.
• You can try proactive namservers (sorry, if that sounds like a pitch, but it’s designed for exactly this sort of situation), I wrote a non-sales-y explanation of it over on CircleId awhile back.
• You can upgrade to Enterprise DNS which adds the fourth anycast cloud to your available nameservers (plus has access to 24×7 support escalation)

Thanks again to DNSMadeEasy and DNSimple, as we’ve commented before, in the DNS biz when there is a DDoS on there are two types of DNS companies, the ones that pick up the phone (or hop on IM) and the ones that fire up the telemarketing crew. We heard from the former overnight, no doubt many of you will be hearing from the latter today.

A big thanks as well, to the fearless sysadmins and intrepid devteam – Erol, Evan, Mike and Brad C, were all up, all night going full tilt and are still at it now. (Can’t wait until we can all get some sleep!)

At the end of the day you have to make the decision that is right for your company or organization. We sincerely regret this incident and we are doing our best to stay in front of this one and get out in front of the next one. There will be a next one, but not only against us, but against the other guys too.

 Discussion

For some reason I can never figure out how to enable comments for these posts, we’ve created a thread on easyCafe to discuss this incident.

I know a lot of you have emailed me personally, I am sorry if I am late in responding or get behind in our threads, I am reading everything that comes in and I may not be able to get back to each of you individually as it’s starting to pile up.