Category: comcast

Mar 10

HBO NOW DNSSEC Misconfiguration Makes Site Unavailable From Comcast Networks (Fixed Now)

Wow! Talking about insanely bad timing…  yesterday at Apple’s big event, HBO announced “HBO NOW”, a new streaming service available for only $15/month that will give you access to all HBO’s content.  This was great news for those people who want to “cut the cord” and not have to pay for a cable TV subscription to get content such as this from HBO. All you had to do was go to order.hbonow.com to get started.

One slight problem – the folks at HBO had signed the hbonow.com domain with DNSSEC, but had not done so correctly!

As a result, the many networks around the world that perform DNSSEC validation to ensure that customers are getting to the correct sites (versus being redirected to bogus sites for phishing or malware) were blocking customers from getting to the possibly bogus order.hbonow.com!

In the text below, I will:

  • Explain what appears to have happened with HBO’s misconfiguration of the hbonow.com domain.
  • Provide two examples confirming what seems to have happened.
  • Speculate on why this occurred.
  • Offer some suggestions on what we need to do next.

Comcast (and Google’s Public DNS) Blocks HBOnow.com

One of those networks here in the USA performing DNSSEC validation was, of course, Comcast (and they have been doing so since January 2012), who is both the largest Internet Service Provider (ISP) in North America and also the largest cable TV provider.  They also own NBCUniversal and create their own video content. So of course the immediate reaction was for people to take to Twitter and blame Comcast:

But here’s the thing:

Comcast was CORRECT in blocking HBO’s site!

Because:

  1. The .COM top-level domain (TLD) had a DS record indicating that the hbonow.com site was signed with DNSSEC.
  2. The hbonow.com domain did NOT have the corresponding DNSKEY record.
  3. Comcast’s DNSSEC-validating DNS resolvers identified the problem and blocked access to the site on the assumption that this could have been an attacker attempting to redirect people to an unsigned and potentially bogus website.

DNSSEC worked correctly to prevent people from going to a bogus site.

Unfortunately, the DNS records were “bogus” not because of an attacker but rather because of a misconfiguration on HBO’s end.

This is not the first time Comcast has dealt with a site with misconfigured DNS records.  If you remember back to 2012 there was the issue with NASA.GOV, which turned out to be a problem with the changing of DNSSEC keys.  Comcast and NASA provided a detailed explanation of what happened then.  (And in another case of spectacularly bad timing, the outage occurred on the day of the SOPA/PIPA website protests, leading then to charges on Twitter that Comcast was deliberately blocking sites.)

UPDATE: I’ve had people tell me they also couldn’t get to the HBO NOW site on networks that use Google’s Public DNS Servers as their DNS resolvers – which makes sense because Google has been performing full DNSSEC validation on sites since May 2013.  (I just did not see anyone tweeting about this…)

HBO’s “Solution”

HBO has seemed to “fix” this issue by, unfortunately, simply removing DNSSEC records and returning the domain to a completely unsigned / unprotected state.   Once the incorrect DNS records age out of DNS resolver caches (based on their Time-To-Live (TTL)), or if the DNS resolver caches are flushed of the current records, then the domain will resolve correctly and people will be able to get to the site.

I’ll speculate on what happened in a moment… but here is some confirmation of what occurred.

Confirmation Using DNSViz

Last night on Twitter, Jason Livingood, VP of Internet Services for Comcast (and also, in full disclosure, a member of the Internet Society Board of Trustees as well as a long-time participant in IETF standards activities), tweeted out a DNSViz analysis of the order.hbonow.com domain (click/tap image for larger version):

DNSViz status of order.hbonow.comIt shows in there that there is a DS record for “hbonow.com” that points to a DNSKEY record with the id 51249.  However, that DNSKEY record does not exist in the actual DNS records for “hbonow.com”.

This is why the failure occurred.

When I look at DNSViz right now for the domain, the picture is different:

DNSViz of order.hbonow.comThere is no DS record for hbonow.com and so there is no DNSSEC failure.

Confirmation Using Dig

On my own home office network, I (of course!) use a DNSSEC-validating resolver and found myself unable to get to the order.hbonow.com site.  Last night when reviewing the news about the Apple event presumably somewhere in there my DNS resolver pulled the DNS records for hbonow.com (perhaps due to web browser “pre-fetching”) and so the old records are in my DNS resolver’s cache.  When I go to a command-line and type “dig +dnssec ds hbonow.com”, I get back the following:

;; QUESTION SECTION:
;hbonow.com. IN DS
;; ANSWER SECTION:
hbonow.com. 10697 IN DS 51249 7 1 90DC90D0578FCFDDF6ED5DE0B35E9652CD2396A8
hbonow.com. 10697 IN RRSIG DS 8 2 86400 20150315045041 20150308044041 13787 com. NAY+BNRi4c6rzLOyoFN4OPOGbbUFuDu/kfO37m00pKkSwXxhAa0qkTTQ HIvzeaFPY54hdJlqH1EzdUEDuL2Nz2stv7iQmsakBaHf3fjHpe2L9H4C Q+wk8yc1vmHdcaUhJyuWYalLwJqg8GWmCXUzWAc6JAoZTPOzF4yZkshp unE=

If you notice the “51249” in the first line of the “ANSWER” section, that matches up with what was shown in the first DNSViz image above as the ID of the DNSKEY that is missing.

When I connect into a system on another and perform the same dig command, I get a different response:

;; QUESTION SECTION:
;hbonow.com. IN ANY

There was no answer section, which means there is no DS record.  If I were on that network I would be able to get to the order.hbonow.com site.

The Time-To-Live (TTL) Issue

If you look back at those responses from my network, you will see the “10697” in the answer section.  This is the number of seconds that the record will remain in the cache of my DNSSEC resolver.  In the time it has taken me to write this post, that number is now down to “5224”  – so about 87 minutes left until that record ages out and my system will stop blocking access to the site.

In Comcast’s case, Jason tweeted out that they flushed the caches on their DNS resolvers and so people should have been able to get to the site right after that.   In my case, I logged into the web admin menu for my home server/gateway and clicked the button to flush the cache… but that didn’t seem to work (and so I’m going to be raising a ticket with the software distribution).

For others out there, they still may be unable to get to order.hbonow.com until the TTL expires in the cache of their DNS resolvers.

Speculation On What Happened

Judging from all of this, here is my guess as to what happened.

1. At some point, HBO signed the domain with DNSSEC. The name server (NS) records indicated that the authoritative DNS servers for hbonow.com are at Dyn’s “DynECT” Managed DNS Service.  DynECT provides a way to very easily sign your domain. I use this service myself for several of my personal domains.  You check a couple of boxes and Dyn takes care of all the signing and re-signing for you.  It has worked well for me.

2. A DS Record was uploaded to the .COM TLD. To tie the domain into the “global chain-of-trust” of DNSSEC, a DS record had to be uploaded to the .COM registry.  This DS record provides a fingerprint (a hash) of the DNSKEY record used to sign the domain.   Unfortunately, right now the only way to transmit a DS record to the .COM registry is through the registrar where you registered the domain name.  I know from personal experience that this involves a manual copy-and-paste of the DS Record from Dyn’s web management interface into my registrar’s web management interface.

Next I think either one of two things happened:

3a. The DNSKEY was updated (rolled over) but the DS record at the .COM registry was NOT updated.  Within DNSSEC, a Key Signing Key (KSK) is set to expire at a certain date and time.  Services operated by DNS operators like Dyn will automagically generate new keys and re-sign the zone. In doing so, the DNS operator will also generate a new DS record.  However, there is no automated way for a DNS operator to get the new DS record to the TLD registry.  (This is something the industry is discussing right now!)

Given that I’m not finding any other DNSSEC-signed records in the cache of my local DNS resolver, I think the answer may more likely be…

3b. HBO decided to remove DNSSEC signing, but forgot to remove the DS record from the .COM registry.  Someone at HBO may have decided to remove the DNSSEC signing from the domain.  Perhaps the signing was just an experiment by someone on the IT team.  Or perhaps someone decided to remove the DNSSEC signatures in advance of the launch so that there was one less variable during the huge launch with Apple.

But… in removing the DNSSEC signatures they forgot to remove the DS record from the .COM registry.

Again, this goes to the manual nature of this process.  Someone could have very easily gone into Dyn’s web management interface and un-checked the box for DNSSEC signing.  Easy. Simple. Done.

But they would also have to login to the registrar’s web management interface and remove the DS record. This may have been the step that someone forgot.

The problem here is that:

  • IF a DS record EXISTS in a TLD;
  • AND the corresponding DNSKEY record does NOT exist in a domain;
  • THEN an attacker could be trying to substitute in an unsigned set of DNS records.

This is exactly the kind of “attack” that DNSSEC is designed to prevent!

Unfortunately, HBO seems to have “attacked themselves” by missing a step in the operations of DNSSEC.

What Next?

This failure last night really speaks to the hole we have in the DNSSEC signing process where there is no easy way for a DNS Operator who is NOT the registrar to update the TLD registry with the new records.  The failure here is because of the manual cut-and-paste process that must be currently used.    I wrote about this in a post:

which in turn pointed over to Olafur Gudmundsson’s post on CloudFlare’s blog:

If we look at the steps of DNSSEC signing today, they look like:

DNSSEC Signing Steps

The challenge we have is to somehow improve the communication between the DNS Operators and the Registries.

In the HBO NOW case, if you go back to my speculation, either of two things could have happened with better automation:

1. The DNS Operator could have updated the registry with a new DS record. If the issue was my “3a” above where there was a key rollover without an update to the DS record, the DNS Operator (Dyn in this case) could have updated the .COM registry with the new DS record.

2. The DNS Operator could have signaled the registry to remove the DS record. If the issue was my “3b” where HBO turned off DNSSEC signing, the DNS operator (Dyn) could have signaled the registry to remove the DS record.

We need to fix this!

We need to have better automation here so that these kind of manual issues do not cause failures in the DNSSEC validation process.

If you’d like to help, there is a public mailing list set up for anyone who is interested.  You can join the effort and subscribe at:

https://elists.isoc.org/mailman/listinfo/dnssec-auto-ds

This work will be ongoing for quite some time and will probably wind up in the DNSOP Working Group within the IETF.  It’s a critically important challenge we need to address to bring further automation to DNSSEC deployment and help many more people secure their domains.

Meanwhile, we’ll have to wait to perhaps get some more official news out of Comcast and or HBO… but this appears to be what happened last night.

Comments, suggestions, feedback or disagreements with my analysis are all welcome as comments!

And… if you want to get started with DNSSEC, please do visit our Start Here page to begin.

Other potential discussions of this post:

Jul 22

Great news! New IPv6 Web Sites: NBC.com, NBCSports.com, UniversalSports.com

Very cool news in Comcast’s announcement today that they launched IPv6 support for:

Naturally I had to check… and sure enough, using the IPvFoo plugin for Chrome, there in the upper right of www.nbc.com…

NBC.com and IPv6

was the green “6″ showing it is connecting over IPv6:

NBC dot com and IPv6

Now, clicking on the green six in my browser bar will show that many of the sites that provide content into that web page are still IPv4-only, but it’s certainly a great start to have the main websites available over IPv6.

Kudos to the teams at Comcast and NBCUniversal for making this happen!

Feb 05

Comcast’s Speedtest Now Breaks Out IPv6 Speed Vs IPv4 Speed

A tip from John Jason Brzowski let us know that Comcast’s Internet speed test at speedtest.comcast.net now performs speed tests over both IPv6 and IPv4 and shows you the results separately.  This is a public test that anyone can use, regardless of whether you are a Comcast customer or not.  Perhaps obviously, for the IPv6 test to work you need to either have native IPv6 connectivity from your ISP or you need to have an IPv6 tunnel for your network.  Without that you’ll just get a regular old IPv4 test.

Naturally I had to try this out and was quite pleased with the results. I am NOT a Comcast customer so the results are for another ISP. I do have native IPv6 connectivity so this was not tunneled traffic. Here was my test yesterday with the closest geographic server (which may or may not relate to network proximity – I didn’t do much checking on that):

Comcast XFinity Speed Test

Of course I was pleased that IPv6 was faster!  I assume this probably had to do with more congestion on the IPv4 network at the precise time I did the test.  As you’ll see below, IPv6 was not always faster.

For those familiar with these type of speed tests, the test performed two separate upload and download cycles for IPv4 and IPv6.  As you can see from the center of the image a cool feature is that you can get a link to an image that you can then share out to social networks or use in other places.  For example, here is the link to my image:

Now, of course I had to try this multiple times during yesterday to see how the results varied – and as is true with pretty much all of these speedtest sites the results DID vary widely.  Some of the results included:

 

I tried other servers in other parts of the US and had similar types of variation.

And then to my amusement I tried the test today shortly before writing this post and found that my speed has degraded significantly. Two results from Boston and one from the New Jersey server:

  

Just to check I tried a couple of other speed test sites and they provided similar results today.  Now the explanation for this drop in my own bandwidth is probably pretty simple.

Snow.

Today we’re experiencing a major snowstorm here in New Hampshire (and all of the northeast USA) and so all the schools are closed and many kids are at home along with parents who need to be home with them.  So people are undoubtedly streaming more movies, playing more online games and just consuming much more online bandwidth than they usually do during the day.  My Internet connection is through my local cable provider… so it’s shared through my neighborhood, and so there we are.  Tomorrow when everyone goes back to school my daytime speed should increase! :-)

All comments about snow aside, this is very cool for Comcast to break out the speeds by protocol this way.  They are of course NOT the only speed test out there that does this.  Other IPv6 vs IPv4 speed tests include sites such as  http://ipv6-test.com/speedtest/  and http://www.speedtest6.com/

Congrats to the team at Comcast for making this available!

P.S. I’d note that Comcast has to be collecting some fascinating measurements out of this effort because they are gathering test results from not only their own customers but also from all of their competitor’s customers who use this test site.  They can then come up with statistics and metrics about the performance of those competitor networks.  A rather brilliant move by someone within Comcast! Now… what would be great for the larger Internet community would be if they could also find some way to perhaps expose some aggregated level of information about what they are are seeing in terms of IPv6 performance across the range of ISPs from people using the site… maybe a topic for a presentation by someone at Comcast at a future event?  (Hint, hint…)

Feb 05

Comcast’s Speedtest Now Breaks Out IPv6 Speed Vs IPv4 Speed

A tip from John Jason Brzowski let us know that Comcast’s Internet speed test at speedtest.comcast.net now performs speed tests over both IPv6 and IPv4 and shows you the results separately.  This is a public test that anyone can use, regardless of whether you are a Comcast customer or not.  Perhaps obviously, for the IPv6 test to work you need to either have native IPv6 connectivity from your ISP or you need to have an IPv6 tunnel for your network.  Without that you’ll just get a regular old IPv4 test.

Naturally I had to try this out and was quite pleased with the results. I am NOT a Comcast customer so the results are for another ISP. I do have native IPv6 connectivity so this was not tunneled traffic. Here was my test yesterday with the closest geographic server (which may or may not relate to network proximity – I didn’t do much checking on that):

Comcast XFinity Speed Test

Of course I was pleased that IPv6 was faster!  I assume this probably had to do with more congestion on the IPv4 network at the precise time I did the test.  As you’ll see below, IPv6 was not always faster.

For those familiar with these type of speed tests, the test performed two separate upload and download cycles for IPv4 and IPv6.  As you can see from the center of the image a cool feature is that you can get a link to an image that you can then share out to social networks or use in other places.  For example, here is the link to my image:

Now, of course I had to try this multiple times during yesterday to see how the results varied – and as is true with pretty much all of these speedtest sites the results DID vary widely.  Some of the results included:

 

I tried other servers in other parts of the US and had similar types of variation.

And then to my amusement I tried the test today shortly before writing this post and found that my speed has degraded significantly. Two results from Boston and one from the New Jersey server:

  

Just to check I tried a couple of other speed test sites and they provided similar results today.  Now the explanation for this drop in my own bandwidth is probably pretty simple.

Snow.

Today we’re experiencing a major snowstorm here in New Hampshire (and all of the northeast USA) and so all the schools are closed and many kids are at home along with parents who need to be home with them.  So people are undoubtedly streaming more movies, playing more online games and just consuming much more online bandwidth than they usually do during the day.  My Internet connection is through my local cable provider… so it’s shared through my neighborhood, and so there we are.  Tomorrow when everyone goes back to school my daytime speed should increase! 🙂

All comments about snow aside, this is very cool for Comcast to break out the speeds by protocol this way.  They are of course NOT the only speed test out there that does this.  Other IPv6 vs IPv4 speed tests include sites such as  http://ipv6-test.com/speedtest/  and http://www.speedtest6.com/

Congrats to the team at Comcast for making this available!

P.S. I’d note that Comcast has to be collecting some fascinating measurements out of this effort because they are gathering test results from not only their own customers but also from all of their competitor’s customers who use this test site.  They can then come up with statistics and metrics about the performance of those competitor networks.  A rather brilliant move by someone within Comcast! Now… what would be great for the larger Internet community would be if they could also find some way to perhaps expose some aggregated level of information about what they are are seeing in terms of IPv6 performance across the range of ISPs from people using the site… maybe a topic for a presentation by someone at Comcast at a future event?  (Hint, hint…)

The post Comcast’s Speedtest Now Breaks Out IPv6 Speed Vs IPv4 Speed appeared first on Internet Society.

Apr 30

Comcast Launches IPv6 Trials For Business Customers – Sign Up Today

comcast business logoWe were very pleased to see news yesterday on Comcast’s corporate blog about the launch of IPv6 services for businesses. Comcast’s John Jason Brzowski wrote there that:

  • Business Ethernet customers have had full IPv6 service and support in place for them since the beginning of 2013.

  • IPv6 trials are about to get underway for our Business Internet customers and we hope to launch full support shortly after completing the trials. Customers interested in signing up to participate in the trials can do so at http://www.comcast6.net/index.php/commercial-broadband-ipv6-form.

It is excellent to see Comcast offering these trials to their business users and we look forward to hearing of the success of those trials and the move to full IPv6 support.  Congrats to the team at Comcast for getting to this point.

If you are a Comcast Business customer, now is a time when you can sign up and get started with ensuring that your networks work fine with IPv6.   Why wait?

Mar 29

Comcast Publishing Domains Failing DNSSEC Via Twitter

Comcast DNS Twitter accountHow do you know when a domain is failing DNSSEC validation? What if there was a way to let the broader industry know about these validation failures?  The folks over at Comcast’s DNS team have been trying an experiment for a while in posting these DNSSEC validation failures publicly to Twitter at:

https://twitter.com/comcastdns

If you are a system/network operator deploying DNSSEC and want to be alerted when sites are found to be failing validation, following this Twitter account is one way you can get alerts.

I don’t know whether publishing domains failing DNSSEC validation via Twitter will really be a long-term solution to letting the wider industry know about domains that are currently failing validation, but I applaud Comcast’s DNS team for trying something different … and I do follow the account myself because I find the occasional tweets interesting to see.

Mar 26

Video: What are “Negative Trust Anchors” for DNSSEC?

What are “negative trust anchors” for DNSSEC? What function do they perform? Why do we need them? In this video, Dan York interviews Jason Livingood about his Internet-Draft on this topic and answers these and other questions:

The Internet-Draft can be found at:

http://tools.ietf.org/html/draft-livingood-negative-trust-anchors

Jason and his co-author are seeking comment and would appreciate feedback from people about this draft – does it make sense? Would you use it? Do you see any ways to improve their ideas? Their email addresses can be found at the end of the document and they are definitely open to feedback.

Jason Livingood is Vice President of Internet & Communications Engineering at Comcast and is one of the co-authors of this draft.

More information about DNSSEC can be found at the Deploy360 website at:
http://www.internetsociety.org/deploy…

This interview was recorded at the 86th meeting of the Internet Engineering Task Force (IETF) in March 2013 in Orlando, Florida, USA.

Apr 26

Comcast Formally Launches IPv6 Home Networking Pilot

This is huge! In a pair of blog posts today Comcast formally launched its IPv6 “Home Networking Pilot”:

As I explained in an earlier post about this impending launch, support for home networks is a critical step for getting IPv6 more widely deployed, particularly as we collectively prepare for World IPv6 Launch on June 6, 2012. As Jason Livingood explains in his post (my emphasis added in bold):

Just as with our standalone computer support for IPv6, customer home networking is also native dual stack. This means that eligible customers will be provisioned with IPv6 addresses in addition to their IPv4 address. We maintain our commitment to the goal of a seamless transition to IPv6 and strongly believe that native dual stack is the best approach for our customers. We also believe that this strategy will over time will meaningfully differentiate our service from our competitors in a way that customers will greatly appreciate. Our native dual stack Xfinity Internet service will provide customers with direct IPv4 and IPv6 access, without the need to use a tunnel, proxy, network address translator, or other inefficient, outdated, and error-prone middlebox. That means customer Internet access will continue to be direct and fast. And because middlebox solutions are not used, customers avoid the risk that certain applications slow down, fail to work, or experience other annoying errors. Since two of the main reasons customers buy our Xfinity Internet service is reliability and the speed — and this approach ensures that we maintain both while other ISPs may face challenges doing so over time — we think our strategic approach to IPv6 will be a winner in the marketplace in the coming years.

John Jason Brzozowski hits a similar note in his post:

Native dual stack support remains central to Comcast deployment of IPv6, which means that customers who are enabled with IPv6 for home networking will be provisioned with IPv6 in addition to IPv4. This approach allows us to avoid the near term use of other types of transition technologies like tunnel and large scale Network Address Translation (NAT). Our experience and industry best practices continue to suggest that native dual stack offers the best path to a seamless IPv6 transition and an optimal customer experience, which is paramount to Comcast.

John Jason also makes a key point that not all home routers currently fully support IPv6 and directs customers to Comcast’s MyDeviceInfo site for the current list of devices Comcast has tested with IPv6.

As Jason Livingood explains, this initial launch will occur in a few of Comcast’s pilot markets but then will be expanded as rapidly as possible to other Comcast subscribers.

For us as an industry this is outstanding news… and many congratulations to Jason, John Jason and their teams!

Now lets see the rest of the ISPs out there join Comcast in providing this level of IPv6 support!

Apr 12

Comcast Enables IPv6 For Xfinity and Xfinity TV

Great news out of Comcast this week related to IPv6 – they have now made two of their major content portals available over IPv6!  From their comcast6.net page on April 10, 2012:

The newest part today moves two of our major portal sites to IPv6, including Xfinity andXfinityTV. This critical move was made possible via close cooperation with Akamai, our CDN vendor. Over time, we will introduce support for native IPv6 for all of our other key websites.

As we get closer to World IPv6 Launch on June 6, 2012, it’s critical to get content (i.e. web sites) available over both IPv4 and IPv6 so that people joining IPv6 networks will be able to natively connect to these sites.

Kudos to Comcast for moving these two large sites over to IPv6 and we look forward to seeing even more of their content moving to IPv6 in the weeks and months ahead.