What We Are Doing About Chronic DDoS Attacks

[ Yesterday we started sending this in an email to all members on the system. We are posting it here for reference.  ]

The number of DDoS attacks directed against the easyDNS system (indeed across all DNS providers) has escalated this year and we wanted to bring you up to speed on measures we are taking to deal with this.

The big problems we face when a domain here is attacked is that we initially have to figure out who the target is, and while the attack is in progress, all the domains sharing the same name server delegations are impacted, sometimes moreso, sometimes not so much depending on the intensity of the attack and the seamlessness with which we can activate mitigation strategies with our mitigation partners.

We are admittedly not satisfied with how the last two DDoS attacks went. The August 12th attack was not very intense and should have been easily handled. It wasn’t and it was the final straw in a string of disappointments here. As a result we’ve made a fundamental change in our mitigation partner strategy and it had already proved worthwhile during the Aug 28th attack. We continue to implement more improvements.

Additionally, starting this fall we will be moving to a new name server numbering scheme, one that will drastically reduce the number of domains that share a particular name server delegation.

This will give us the ability to instantly identify the target of a given DDoS and at the same time, limit the impact of the DDoS to the targeted domain(s).

Doing this requires some re-engineering on our part and the last of those updates are scheduled to be completed this fall.

We will send another email when we are ready to implement the new naming structure.

In the meantime, if you operate a mission critical domain which must have 24×7 DNS availability at all times, please read the following carefully:

The official easyDNS “best practice” advice for guaranteeing DNS availability for mission critical domains:

Use multiple DNS solutions. 

While most DNS providers guarantee you that they never experience downtime, the reality is that we all do to some degree from time to time, so if you absolutely have to be online, all the time, use a combination of multiple DNS providers.

Using a single out-of-band name server in your delegation increases your redundancy by an order of magnitude.

We wrote extensively on this subject in “DNS and DDoS Attacks: How To Stay Up When Your DNS Goes Down” (also see the “Further Reading” section at the end of this email).

Tools you can use to make this easier:

easyRoute53: This is an integration layer to Amazon’s Route 53 DNS, through this you can use the same control panel here to control your DNS settings both here and on Amazon Route 53. It’s set-and-forget as easyRoute53 automatically pushes your DNS updates out to the Route 53 system (it can also be configured to load your settings from Route 53).

You can use your own Amazon AWS keys or if you don’t want the hassle of setting that up, we have just launched a fully managed version of Route 53 where we take care of everything for you. Single point of billing, single point of contact, single control panel but dual provider redundancy.

One main advantage of the Fully Managed option is that we are your support for Route53, you don’t need to purchase an expensive support incident or contact from Amazon.

Whether you go with a self-serve or fully managed Route 53 backup for your DNS, the end result is that you’ll have dual provider redundancy and your DNS will be on 7 or 8 anycast clouds spanning over 40 global name servers – that’s more DNS redundancy than a lot of top level domains.

External zone transfers: 

You can use any external name server or DNS host that will allow third-party masters to do zone transfers from us, as outlined here:

http://helpwiki.easydns.com/index.php/Enabling_External_Zone_Transfers

We find our systems play well with any bind name server, recommended external providers are DNSMadeEasy (no relation) and No-IP.

Proactive Nameservers:

Proactive Nameservers is best thought of as “Hot Swappable name servers” or failover at the name server level. This system monitors your existing name server delegation and when it finds a problem it automatically switches you to the most optimal name server delegation. We need to be your registrar for this to work and you can learn more about our patent pending technology at http://proactivenameservers.com. Our customers who started using Proactive Nameservers have come through every DDoS attack since with flying colours.

Conclusion

Over the years we feel our job has evolved. It is no longer just about “selling you DNS services”. If you are running a mission critical domain, something that holds up a piece of the internet or that many stakeholders rely on, then that isn’t enough anymore. We have to facilitate the tools and the methodologies to take your DNS redundancy up a level so that you can stay ahead of the threat vectors you face today.

We can b.s. you and give you a string of excuses on why these incidents were flukes and that we’ll never again have DDoS related impact because of this or that improvement.

Or we can make those same improvements and tell you the truth: this is an arms race. There is always a more sophisticated attack vector or a bigger botnet. On really lousy days there is both.

The best practice of achieving DNS uptime via redundancy has moved way beyond “having at least two name servers on disparate networks” or even outsourcing it to a single DNS provider. You have to view each DNS provider, each network or solution as a possible Single-Point-of-Failure unto itself. It still astounds me to this day when I see multi-billion dollar companies or essential public infrastructure services invest hundreds of thousands or even millions of dollars into load balancers, monitors, failover devices, firewalls, security audits, penetration testing and disaster recovery plans and the whole enterprise is held up by a couple of name servers and there is no backup plan for “what happens if the DNS fails?” This is odd considering it’s one of the most popular attack vectors in-play today.

While we are continuing to make additional changes that will drastically reduce the impact of future DDoS attacks, our opinion is that the Best Practice path to acheiving 100% DNS availability is through having multiple DNS solutions and using a coherent system to manage those solutions and keep them in sync.

Your mission critical servers probably have redundant power supplies, and RAID arrays, today’s environment necessitates an analogous approach to DNS.

As always, please feel free to call or email me personally with any concerns or feedback. I apologize in advance if I cannot reply to every comment individually, but I do read every single one of them.

Thank you for choosing easyDNS.