What do you do if you go to start live video streaming of an event - and the streaming service you use won't start??? How do let people waiting to watch know? How do you fall back to another service?
Last month at the IETF 90 meeting in Toronto I experienced this nightmare of anyone doing live video streaming across the Internet. We had a presentation at noon on Thursday that we had widely publicized in social media and via email. The cameras were all set up. The producer was all ready to do the switching and encoding out to YouTube. Everything was good to go.
Then at 11:45am, I went to do my part in the process. I needed to login to the IETF YouTube account and basically click the "Start Streaming" button inside the "live event". At that point the YouTube servers would accept the encoded stream from the producer's gear and the stream would go live.
BUT... instead I got this message in a bright red bar across the top:
The message read:
YouTube live is undergoing maintenance. Events cannot be created, started or stopped. Events already started will continue to stream uninterrupted.
Yes... we had no way to START the live video stream!
Over the next few minutes I kept refreshing that page but the warning stayed up there. I was getting rather nervous as the launch time approached!
Now, as it happened, we had already planned to have a second live video steam going out for the event through a different service, Livestream.com, for a different reason.
Why The IETF Uses YouTube For Live Video Streaming
To explain a bit more, we have been using Google's YouTube live video streaming for some of the larger IETF plenary sessions for five reasons:1. The live video stream is available over both IPv4 and IPv6.
2. The stream works easily across pretty much all desktop and mobile devices.
3. Google's live streaming infrastructure seems able to scale to whatever capacity is needed.
4. The recording of the stream is immediately available after the event in the IETF's YouTube channel.
5. There is no cost for using the service beyond our local costs to produce the content (and no infrastructure that the IETF itself has to maintain).
Of these, really the most critical reason for using YouTube live streaming is the first - that it streams out over IPv6.
The IETF is the organization behind the IPv6 specification and has declared that all new IETF standards need to incorporate IPv6. Therefore in the spirit of "eating your own dog food" the IETF tries to use services that work over IPv6 whenever possible. Other live video streaming services have met the reasons 2-5 above, but not the #1 reason of working over IPv6.
We have specifically been using what Google used to call "YouTube Live" but now seems to just be calling "YouTube live events" versus Google's newer "Hangouts On Air (HOA)". These YouTube live events are events you schedule in advance and can use with advanced video encoders. An advantage is that these events provide streaming configuration info that I can provide in advance to the company running the audio and video at the event so that they can be prepared in advance. YouTube also helpfully provides a countdown timer for people visiting the event page. We haven't switched to using HOAs because they haven't yet provided the advance configuration information we want.
Anyway it has all worked well for live streaming out plenary sessions for a couple of years now.
Google Doesn't Live Stream Into Germany
However, as we discovered again that week.... Google will not stream live video into Germany! It seems Google has a legal dispute with a German intellectual property rights organization (GEMA) and Google has decided that rather than run into trouble with GEMA they will simply NOT allow live streaming into Germany.
So, alerted to this issue by some IETF remote participants in Germany who were unable to watch the live video streams of the technical plenary earlier in the week, we had arranged to also stream this Thursday session out over the Internet Society's Livestream.com account. Now, unfortunately it would not be available over IPv6 because Livestream.com still only works on legacy IPv4 networks, but Livestream.com did not have the streaming restrictions Google had and so at least people could view the stream in Germany. As a bonus, all the "subscribers" to the Internet Society's Livestream.com channel would also get notified and potentially be able to watch the stream - but the primary reason was so that people in Germany could watch the video stream.
The great thing about IETF meetings is that a massive amount of Internet connectivity is brought into the meeting hotel (because you have 1,200+ engineers who do most of their work across the Internet!) and so there are NO bandwidth problems for streaming. We could probably stream out to a dozen different live streaming services simultaneously if we set up our local software/equipment to do so.
Making The Alternate Stream The Primary Stream
The good news for us was that this "alternative" live video stream set up purely for viewers in Germany could now become the primary video stream. I rapidly updated the Google+ "event page" for this session to note the new URL for streaming and we spread the word through IETF social media channels and email lists. It wasn't 100% seamless but we were able to get people watching the live video stream.
We were also able to direct people to some of the other IETF remote meeting participation mechanisms, including audio streaming and a conferencing system called "Meetecho" that streamed the slides and lower webcam-quality video.
Throughout the hour-long event I kept checking the Live Control Room inside of YouTube to see if we could start the original stream, but we were never able to do so. A couple of times the red warning box went away, but we could not establish a connection from YouTube's streaming service to our equipment on the ground there in Toronto. Finally, as the time went on it became clear that the connection wasn't going to happen and so I just gave up trying.
The good news is that the producer was also making a local copy of the stream that we would be able to upload later to the IETF's YouTube channel.
Lessons Learned
I took away from this experience three primary lessons for all future live streaming sessions. Do note, too, that I think of these as generic lessons for all live streaming services and events. It happens that this time the failure was with Google's YouTube live events service, but the failure could have been with Livestream.com, Google's Hangouts On Air, Ustream or any of the many other live video streaming services out there.
1. Always Promote An Event Page Separate From The Streaming Service
We were able to rapidly redirect people to the new location of the live video stream in large part because we had been promoting the Google+ event page as the place to go to watch the live stream. We had promoted this on the IETF's Twitter account, Facebook page, Google+ page and also over various IETF email lists and on various other websites. All the promotion pointed people to this page.
So the good news was that all we had to do was update this page with the new info and people could switch over to watch the new stream.
We had NOT been promoting the direct YouTube link for the stream. Had we done so, we could have still updated the page through editing the description of the YouTube video and/or leaving comments - but it would not have necessarily been as easy for visitors to see.
Promoting a separate page was a deliberate choice I made based on some previous bad experiences with live streaming where I had to stop a streaming session and restart with a brand new URL. For that reason I've been promoting a separate page.
In fact, for the IETF Plenary sessions, we've been promoting a separate page under IETF control on the IETF website - http://www.ietf.org/live/ - where we can embed the live stream video and also keep the page updated. At the IETF meeting it is possible for me or someone else to easily go in and update that page. Plus it is a very simple URL that we can promote widely.
I don't honestly remember why we didn't use the www.ietf.org/live/ page to stream out this Thursday morning sponsor presentation other than that the decision to live stream the session happened the day before and for whatever reason we went with a Google+ Event page as the page to promote.
Next time we'll probably promote the www.ietf.org/live/ page.
The key point is that you have a page separate from the live streaming service where you can post updates.
2. Have An Alternate Live Stream Either Active Or Ready To Go
As I mentioned previously, in this case we happened to be set up with a second live stream out through Livestream.com purely because we wanted to test the live streaming into Germany. Had remote IETF participants in Germany not asked about this after being unable to view the earlier technical plenary, we wouldn't have had this second stream active.
Next time, we will have a second live streaming service either active or at least on standby ready to go.
At the IETF meetings, we have the luxury of having an insane amount of bandwidth and so there are not the typical connectivity constraints you find in meeting venues. The software and equipment our producer was using could go out to multiple live streaming services. There is really no reason we can't run multiple streams.
For the IETF we still have the IPv6 requirement, which unfortunately Livestream.com does not yet meet. However, it occurred to us after the session that we could have streamed to a Google+ Hangout On Air (HOA) as that would have also streamed out over IPv6 in addition to IPv4. Of course, that would mean relying on two Google services and so you run the risk of having the technical issues affecting one live streaming service also affecting the other - plus there was the whole "streaming into Germany" thing.
We'll definitely keep investigating what other live streaming services may work over IPv6. There are a good number of live video streaming services out there and the number seems to be growing. The company producing the video stream for us also had their own streaming server that we might be able to use as a backup, too. And, yes, we can also have an IPv4-only streaming service available if everything else falls through.
Now, in non-IETF environments where I do have to worry about bandwidth constraints, I will at least have a plan for how I can rapidly spin up a second stream if the first one fails. That's really the key point. What is Plan B and how fast can you make it happen?
3. Have Access To Relevant Social Media Accounts And Other Methods Of Letting People Know
This is perhaps a subset of Lesson #1, but another critical part of our success in redirecting people to the second live stream was that we had access to the relevant social media accounts and other means of spreading the word. I had access to the IETF Google+ page and could make the updates there. Someone else was able to send out a tweet with the new link to the live stream. An email was sent out to all attendees and to other relevant email lists letting them know about the link.
The key point is that when we updated the event page with the new information, we could let people know!
In The End...
... the session was streamed live across the Internet. It was recorded and made available for later viewing. And... we learned a few lessons to make sure our live streaming infrastructure is more resilient next time so that this potential "nightmare" becomes nothing more than just a minor bump and redirection.
What about you? If you do live video streaming what steps have you taken to ensure you can keep streaming in cases like this?
I also recorded an audio commentary about this situation:
If you found this post interesting or useful, please consider either:
- following me on Twitter;
- adding me to a circle on Google+;
- following me on App.net;
- subscribing to my email newsletter; or
- subscribing to the RSS feed.
