Just a guy in Vermont trying to connect all the dots...
Author's posts
Sep 03
The Live Video Streaming Nightmare: What Do You Do When YouTube Won’t Start? (And Lessons Learned)
What do you do if you go to start live video streaming of an event - and the streaming service you use won't start??? How do let people waiting to watch know? How do you fall back to another service?
Last month at the IETF 90 meeting in Toronto I experienced this nightmare of anyone doing live video streaming across the Internet. We had a presentation at noon on Thursday that we had widely publicized in social media and via email. The cameras were all set up. The producer was all ready to do the switching and encoding out to YouTube. Everything was good to go.
Then at 11:45am, I went to do my part in the process. I needed to login to the IETF YouTube account and basically click the "Start Streaming" button inside the "live event". At that point the YouTube servers would accept the encoded stream from the producer's gear and the stream would go live.
BUT... instead I got this message in a bright red bar across the top:
The message read:
YouTube live is undergoing maintenance. Events cannot be created, started or stopped. Events already started will continue to stream uninterrupted.
Yes... we had no way to START the live video stream!
Over the next few minutes I kept refreshing that page but the warning stayed up there. I was getting rather nervous as the launch time approached!
Now, as it happened, we had already planned to have a second live video steam going out for the event through a different service, Livestream.com, for a different reason.
Why The IETF Uses YouTube For Live Video Streaming
To explain a bit more, we have been using Google's YouTube live video streaming for some of the larger IETF plenary sessions for five reasons:1. The live video stream is available over both IPv4 and IPv6.
2. The stream works easily across pretty much all desktop and mobile devices.
3. Google's live streaming infrastructure seems able to scale to whatever capacity is needed.
4. The recording of the stream is immediately available after the event in the IETF's YouTube channel.
5. There is no cost for using the service beyond our local costs to produce the content (and no infrastructure that the IETF itself has to maintain).
Of these, really the most critical reason for using YouTube live streaming is the first - that it streams out over IPv6.
The IETF is the organization behind the IPv6 specification and has declared that all new IETF standards need to incorporate IPv6. Therefore in the spirit of "eating your own dog food" the IETF tries to use services that work over IPv6 whenever possible. Other live video streaming services have met the reasons 2-5 above, but not the #1 reason of working over IPv6.
We have specifically been using what Google used to call "YouTube Live" but now seems to just be calling "YouTube live events" versus Google's newer "Hangouts On Air (HOA)". These YouTube live events are events you schedule in advance and can use with advanced video encoders. An advantage is that these events provide streaming configuration info that I can provide in advance to the company running the audio and video at the event so that they can be prepared in advance. YouTube also helpfully provides a countdown timer for people visiting the event page. We haven't switched to using HOAs because they haven't yet provided the advance configuration information we want.
Anyway it has all worked well for live streaming out plenary sessions for a couple of years now.
Google Doesn't Live Stream Into Germany
However, as we discovered again that week.... Google will not stream live video into Germany! It seems Google has a legal dispute with a German intellectual property rights organization (GEMA) and Google has decided that rather than run into trouble with GEMA they will simply NOT allow live streaming into Germany.
So, alerted to this issue by some IETF remote participants in Germany who were unable to watch the live video streams of the technical plenary earlier in the week, we had arranged to also stream this Thursday session out over the Internet Society's Livestream.com account. Now, unfortunately it would not be available over IPv6 because Livestream.com still only works on legacy IPv4 networks, but Livestream.com did not have the streaming restrictions Google had and so at least people could view the stream in Germany. As a bonus, all the "subscribers" to the Internet Society's Livestream.com channel would also get notified and potentially be able to watch the stream - but the primary reason was so that people in Germany could watch the video stream.
The great thing about IETF meetings is that a massive amount of Internet connectivity is brought into the meeting hotel (because you have 1,200+ engineers who do most of their work across the Internet!) and so there are NO bandwidth problems for streaming. We could probably stream out to a dozen different live streaming services simultaneously if we set up our local software/equipment to do so.
Making The Alternate Stream The Primary Stream
The good news for us was that this "alternative" live video stream set up purely for viewers in Germany could now become the primary video stream. I rapidly updated the Google+ "event page" for this session to note the new URL for streaming and we spread the word through IETF social media channels and email lists. It wasn't 100% seamless but we were able to get people watching the live video stream.
We were also able to direct people to some of the other IETF remote meeting participation mechanisms, including audio streaming and a conferencing system called "Meetecho" that streamed the slides and lower webcam-quality video.
Throughout the hour-long event I kept checking the Live Control Room inside of YouTube to see if we could start the original stream, but we were never able to do so. A couple of times the red warning box went away, but we could not establish a connection from YouTube's streaming service to our equipment on the ground there in Toronto. Finally, as the time went on it became clear that the connection wasn't going to happen and so I just gave up trying.
The good news is that the producer was also making a local copy of the stream that we would be able to upload later to the IETF's YouTube channel.
Lessons Learned
I took away from this experience three primary lessons for all future live streaming sessions. Do note, too, that I think of these as generic lessons for all live streaming services and events. It happens that this time the failure was with Google's YouTube live events service, but the failure could have been with Livestream.com, Google's Hangouts On Air, Ustream or any of the many other live video streaming services out there.
1. Always Promote An Event Page Separate From The Streaming Service
We were able to rapidly redirect people to the new location of the live video stream in large part because we had been promoting the Google+ event page as the place to go to watch the live stream. We had promoted this on the IETF's Twitter account, Facebook page, Google+ page and also over various IETF email lists and on various other websites. All the promotion pointed people to this page.
So the good news was that all we had to do was update this page with the new info and people could switch over to watch the new stream.
We had NOT been promoting the direct YouTube link for the stream. Had we done so, we could have still updated the page through editing the description of the YouTube video and/or leaving comments - but it would not have necessarily been as easy for visitors to see.
Promoting a separate page was a deliberate choice I made based on some previous bad experiences with live streaming where I had to stop a streaming session and restart with a brand new URL. For that reason I've been promoting a separate page.
In fact, for the IETF Plenary sessions, we've been promoting a separate page under IETF control on the IETF website - http://www.ietf.org/live/ - where we can embed the live stream video and also keep the page updated. At the IETF meeting it is possible for me or someone else to easily go in and update that page. Plus it is a very simple URL that we can promote widely.
I don't honestly remember why we didn't use the www.ietf.org/live/ page to stream out this Thursday morning sponsor presentation other than that the decision to live stream the session happened the day before and for whatever reason we went with a Google+ Event page as the page to promote.
Next time we'll probably promote the www.ietf.org/live/ page.
The key point is that you have a page separate from the live streaming service where you can post updates.
2. Have An Alternate Live Stream Either Active Or Ready To Go
As I mentioned previously, in this case we happened to be set up with a second live stream out through Livestream.com purely because we wanted to test the live streaming into Germany. Had remote IETF participants in Germany not asked about this after being unable to view the earlier technical plenary, we wouldn't have had this second stream active.
Next time, we will have a second live streaming service either active or at least on standby ready to go.
At the IETF meetings, we have the luxury of having an insane amount of bandwidth and so there are not the typical connectivity constraints you find in meeting venues. The software and equipment our producer was using could go out to multiple live streaming services. There is really no reason we can't run multiple streams.
For the IETF we still have the IPv6 requirement, which unfortunately Livestream.com does not yet meet. However, it occurred to us after the session that we could have streamed to a Google+ Hangout On Air (HOA) as that would have also streamed out over IPv6 in addition to IPv4. Of course, that would mean relying on two Google services and so you run the risk of having the technical issues affecting one live streaming service also affecting the other - plus there was the whole "streaming into Germany" thing.
We'll definitely keep investigating what other live streaming services may work over IPv6. There are a good number of live video streaming services out there and the number seems to be growing. The company producing the video stream for us also had their own streaming server that we might be able to use as a backup, too. And, yes, we can also have an IPv4-only streaming service available if everything else falls through.
Now, in non-IETF environments where I do have to worry about bandwidth constraints, I will at least have a plan for how I can rapidly spin up a second stream if the first one fails. That's really the key point. What is Plan B and how fast can you make it happen?
3. Have Access To Relevant Social Media Accounts And Other Methods Of Letting People Know
This is perhaps a subset of Lesson #1, but another critical part of our success in redirecting people to the second live stream was that we had access to the relevant social media accounts and other means of spreading the word. I had access to the IETF Google+ page and could make the updates there. Someone else was able to send out a tweet with the new link to the live stream. An email was sent out to all attendees and to other relevant email lists letting them know about the link.
The key point is that when we updated the event page with the new information, we could let people know!
In The End...
... the session was streamed live across the Internet. It was recorded and made available for later viewing. And... we learned a few lessons to make sure our live streaming infrastructure is more resilient next time so that this potential "nightmare" becomes nothing more than just a minor bump and redirection.
What about you? If you do live video streaming what steps have you taken to ensure you can keep streaming in cases like this?
I also recorded an audio commentary about this situation:
If you found this post interesting or useful, please consider either:
- following me on Twitter;
- adding me to a circle on Google+;
- following me on App.net;
- subscribing to my email newsletter; or
- subscribing to the RSS feed.
Sep 02
Watch the 2014 Internet Governance Forum (#IGF2014) Live All This Week from Istanbul (Featured Blog)
Sep 02
Learn IPv6 Basics in VMware #vBrownBag Webinar On Sept 3
Interested in learning more about “IPv6 for IPv4 Brains”? By way of a tweet we learned that Jeff Carrell will be the guest on VMware’s “#vBrownBag” webinar/podcast on this Wednesday, September 3, 2014, at 19:30 US Central Time (which I calculate to be 00:30 UTC on Thursday, Sept 4, if I do the math correctly).
In another tweet Jeff said that he’ll be talking about “IPv6 basics, contrasted to IPv4 (where possible) to assist in the learning curve“. On VMware’s #vBrownBag page there is a link to register for the event.
I don’t know much about VMware’s webinar series, nor have I personally seen Jeff in action providing training, but he’s one of the most active people I’ve seen on Twitter covering IPv6 issues. I’m not going to be able to join in to tomorrow night’s session live, but I’m looking forward to listening to the archive once it is available.
And… if you are inspired after listening to the session to do more with IPv6, please do visit our Start Here page to find IPv6-related resources tailored to your type of organization or role. The time for IPv6 is NOW!
Sep 02
DNSSEC Deployment Maps Updated Today (2 Sep 2014) With Latest New GTLDs
As part of our ongoing publishing of DNSSEC deployment maps showing visually the DNSSEC status of country-code top-level domains (ccTLDs), we also publish a comma-separated value (CSV) file that contains the DNSSEC status of all the generic top-level domains (gTLDs), including all the “new gTLDs” that have been published as part of ICANN’s program. We do not yet have any visual display of the generic TLDs, although that is something we would like to eventually include in the distribution. For the moment they are just listed in a CSV file – and primarily what we are listing is that they have a DS record in the root zone of DNS.
The latest maps and CSV files as of today, September 2, 2014, can now be found online. These maps and CSV files are published every Monday morning with whatever updates have been made to the database. If you would like to receive the maps and CSV files, please subscribe to the dnssec-maps mailing list. The list is “announce-only”, i.e. it is not a discussion list, and so you will only receive the weekly map updates and very occasional administrative updates. You can visit the list archives to see the type of messages sent to the list.
Sep 02
Lifehacker: Disabling IPv6 Will NOT Speed Up Your Internet Connection
IPv6 is not exactly a topic we would normally expect to see on the hugely popular Lifehacker website, but there it was on Friday in a story by Patrick Allan, “Why Disabling IPv6 Won’t Speed Up Your Internet Connection“. In the article, Patrick references an earlier article on How-To Geek that aims to dispel the idea that disabling IPv6 would somehow speed up your Internet access.
In both articles, the authors make the point that earlier versions of web browsers would sometimes have significant delays in connecting to web sites if IPv6 was involved. For example, if a website had a AAAA record with an IPv6 address a browser would sometimes try and try… and try… to connect to the site over IPv6… even if the computer itself had no IPv6 connection!
Thankfully, in 2014 almost all web browsers now use some variation of the “Happy Eyeballs” algorithm defined in RFC 6555 where they try to connect over both IPv6 and IPv4 and use whichever connection works best and so these web browser problems are now a thing of the past.
As both articles note, you may do more damage to your computer by disabling IPv6 these days, simply because we are running out of IPv4 addresses and many sites and networks are already well underway with their transitions to IPv6 (or have already completed those transitions).
We’re very pleased to see Lifehacker pointing this out… and look forward to more such articles as the IPv6 transition is happening rapidly around us!
If you would like to make the transition to IPv6, please see our Start Here page to find resources that will help you out!
Sep 02
Can You Please Help The Ottawa Linux Symposium?
If you have ever used the Linux operating system, could you please help out the Ottawa Linux Symposium (OLS)? For many years OLS has been one of the key events that has helped bring together people from all across the Linux community, and the connections made at OLS have helped to make the Linux operating system that much more powerful and useful. But… as organizer Andrew Hutton recounts on the OLS Indiegogo page, the event has fallen in a bit of a financial crunch and it is now not clear if there will be an OLS in 2015… or ever again.
Could you spare $10? $25? or even $50 or $100? (Or more?)
If so, please help fund OLS on the IndieGogo page!
I first attended OLS back in the early 2000s when I was living right there in Ottawa and working for first a startup called e-smith and then subsequently Mitel Networks. In looking at my list of presentations I can see that I spoke there several times… and the topics I covered take me back to a much different time:
- 2004 OLS – Tutorial: Introduction to OpenPGP, GnuPG and the Web of Trust
- 2002 OLS – Tutorial – Single Source Publishing Using DocBook XML
- 2001 OLS – Maximizing Your Use of CVS
I still remember OLS as the incredibly passionate place where people connected…. and where I made so many connections and learned an amazing amount about Linux.
If OLS was ever important to you… or if Linux has been important to you… please consider donating to help the OLS organization get out of its financial hole and get moving ahead in future years. Organizer Andrew Hutton has poured his heart and soul – and personal money – into making OLS the incredible event it has been… now it would be great if we all can help him! Please consider donating!
Here are a few other viewpoints on the importance of OLS:
- Jon “maddog” Hall: “Ottawa Linux Symposium: May get by with a little help from its friends”
- Matt Domsch: “Ottawa Linux Symposium needs your help”
Please do donate if you can! THANK YOU!
Sep 02
50% Off All O’Reilly Ebooks Through Sept 9, 2014
Want to buy “Migrating Applications to IPv6” at 50% off? (in ebook form) Or any of 8,000 other ebooks and videos?
The folks at O’Reilly are having a “Back to (Tech) School Sale” that is in effect until 5:00pm U.S. Pacific Daylight Time on September 9, 2014. All you need to do is use the discount code “B2S4″ when you are checking out.
I would STRONGLY encourage you to buy my book directly through O’Reilly when you can because you get:
- DRM-free – you can put it on as many devices as you want, and it’s yours.
- As many formats as you want – EPUB, Mobi, PDF, whatever…
- Notification of (free) updates – whenever I post updates to the book, you will be notified.
It’s definitely worth it… I’m buying my tech ebooks directly from O’Reilly whenever I can now, simply because of these three points!
Thank you for considering the purchase of this book – and please do let me know if you have any suggestions or feedback to help make it better. The entire point of the book was to help application developers be able to make their application work over IPv6… so I want to hear from you about how this book can help!
Sep 01
FIR #771 – 9/1/14 – For Immediate Release
Aug 28
Watch LIVE Now! AfPIF Lightning Talks On IXP Toolkit, Deploy360, More
Right NOW in Dakar, Senegal, the last session of the African Peering and Interconnection Forum (AfPIF) 2014 is happening. As we wrote previously, our Chris Grundemann is there – and he will actually be speaking during this “Lightning Talk” session about what we are doing here with Deploy360 in general and in particular with our Best Current Operational Practices (BCOP) efforts. You can watch live at:
http://new.livestream.com/internetsociety/AfPIF2014Day3
It will also be archived for later viewing.
Right now our Internet Society colleague Jane Coffin is talking about the IXP Toolkit that has been developed to help with the creation of Internet Exchange Points (IXPs) (see our IXP page for our resources to help).
The list of speakers will be:
- IXP Toolkit (Jane Coffin, ISOC)
- IPv6 Webinars (Kevin Chege, ISOC)
- Peering in Kenya (Barry Macharia, KIXP)
- The successful implementation of an IXP and its evolution (Adelard Kenese, Burundix)
- DO Base (Chris Grunderman, ISOC)
- Business Intelligence Application (Elaud Kutiwa)
Each is to do a short 5-10 minute presentation.
If you want to see more, you can watch live RIGHT NOW! :-)
Aug 26