I Was Told There Would Be No Math
Once again, my job continues to require the use of math. It’s basic math, but it’s still math. I thought my job was cool, but math is not cool. Hmmmm….
Here’s a word problem for you:
Billy has a band. Billy’s buddy Tommy also has a band, and the two want to get their bands to play together AT THE SAME TIME, but their bands are 20 miles apart. Joey hears about this and says he wants his band to join the club, too, but Joey’s band is another 20 miles in the opposite direction. It just so happens that Billy has a friend named Max, and Max has some crazy voodoo technology stuff so that Billy’s friends’ bands can all play together. But even with this voodoo stuff, if Billy’s and Tommy’s bands are 20 miles apart, and Billy’s and Joey’s bands are 20 miles apart, and Joey’s and Tommy’s bands are 40 miles apart, how bad will the transmission delay be between all the bands? Will it be like middle school recorder concert bad or will it only be garage band bad?
This was our challenge for the week. We knew it was coming, and if you read into my post about the London’s a couple weeks ago you might have seen this coming, too. This week we synched up all three of our campuses to do a multi-campus medley. It was a vision casting Sunday, and the theme was together so the medley featured songs with the word “Together” in the title. I’m not sure “medley” is completely accurate because each of our stages did a snippet of a song with a beginning and end. The songs didn’t necessarily seamlessly flow into one another, although, there was a definite flow. We had Browns Bridge starting off with “Happy Together” as made famous by the Turtles. Our West Auditorium followed with “Come Together” as made famous by the greatest rock and roll band of all time. Buckhead slid in next with “Let’s Stay Together” made famous by the Reverend Al Green. Then finally our East auditorium brought “Join Together” as made famous by the Who. But who would be satisfied with just that. Then we had all four stages join together to finish off “Join Together”. 4 bands on 3 campuses in 3 different cities. So how’d we do it?
Like I said, we knew this was coming when the decision to connect all our campuses together happened last year so this scenario was built into the system when I designed the program for the Londons. However, even with the infrastructure in place there were still some variables to figure out. First up was figuring out the latency in transmission between campuses. I’ll try not to melt your brain with this stuff because it can get crazy if you think too hard about it.
We did an initial test about four weeks ago where we simply fed some music to our Browns Bridge campus and returned it to North Point. We recorded both feeds in Pro Tools and measured an initial round-trip time of 180 milliseconds. The good news was we had some stuff configured wrong on our end and were able to knock that time down to about 92 milliseconds round-trip between our campuses. North Point serves as the hub for our system so this gave us about a 46 millisecond transmission time to our other campuses. 46 milliseconds may seem like a short amount of time–it’s barely more than a frame of video–but in musical terms, it’s quite noticeable. 46 milliseconds is an echo.
Think in terms of how this would play in the context of our performance this morning. A master click track would start on the East stage at North Point and transmit to campuses. The click would arrive at Browns Bridge 46 milliseconds later cueing Browns Bridge to start playing. From the Browns Bridge perspective, they would be right in time with North Point.
Back at North Point the return of the Browns Bridge performance would happen after another 46 milliseconds. In the ears of NP musicians, they would have the click playing BUT they wouldn’t hear Browns Bridge until 92 milliseconds after the initial click. So Browns Bridge is 90ms late relative to North Point at North Point.
In this initial scenario, Buckhead would hear the same thing North Point hears. Browns Bridge would be 92 milliseconds behind the click due to the 46 millisecond travel time from BBCC to NP and then another 46 to Buckhead. So Browns Bridge is 92ms late relative to Buckhead at Buckhead. When the NP West auditorium starts, however, they would appear to be right in time at Buckhead since they would follow the click already being sent from NP. Are you following so far?
One of our volunteers, Chris Lawton, had a good suggestion for us. Chris worked in radio for a long time, and they used to do these kind of link-ups all the time. His suggestion was to delay the click to our local musicians. After running some scenarios, we settled on delaying the click to our local NP musicians one transmission length or 46 millisecond to distribute the delay a little better across campuses. There was nothing we could do about the time between Buckhead and Browns Bridge, but by doing this we could tighten things up at NP.
So here’s where we landed. Click starts at NP and is sent to campuses. 46 milliseconds later BBCC hears it. At that same moment, our local musicians at NP hear the same click so now the return from BBCC is only 46 milliseconds late instead of 92. Of course, North Point now seems to arrive 46 milliseconds late at Browns Bridge. If we were to delay the NP click locally the full round-trip length of 92ms, we would have fixed all the timing problems at NP, however, our campuses would have suffered the same as NP did in the initial scenario which in our world improves nothing.
Of course, the fact that there would always be transmission delay created a musical challenges. To counter this, our music folks devised a pretty simple solution of sustained endings on the tail of each song in the medley to smooth over the transition from one campus to another as much as possible; I can’t speak for our other campuses, but it felt pretty natural in our room. At the end of the medley when all campuses joined in on “Join Together”, each campus simply took their local stage and dropped anything remotely transmitted while our video folks still switched between campuses. Locally at North Point where we have two auditoriums we were able to pull up the opposite auditorium or at a minimum some musical elements since there were no timing problems between the two.
There was still a bit more to this in getting all the bussing right, but I’ll save that for next time…
Currently Reading:
Previous Post
Next Post
That is a really cool idea. I had a question though about how you actually did it (and you said you will have other posts, so if you answer these questions later, I will be fine with it). The click was set on a delay and I was wondering how you did that, with the London program or the Venue console? And also I was wondering if you came off of a split to another console to take in the other campuses or relied on a mix in from that location’s console?
Good questions, Adam. We have a Pro Tools rig on stage next to the drummer for clicks, loops, and any other musical elements that go beyond the band (ex. strings, ebows, etc.). For the medley, there were two click tracks in Pro Tools: one for campuses, and one for local use. The music folks just added a delay plugin to the local click track in Pro Tools to delay it for our stages.
Each campus was sending us a 2-mix of their band, and I’ll probably go into that a little more in the next post.
How did that work when the campuses’ service times didn’t line up? (NPCC has a 12:45 service, but Buckhead has a 6:00 service, etc.)?
Everybody stuck around to do the 12:45. Buckhead did all the songs in the melody themselves at their 6pm.
I am curious about the measured delay.
My understanding from some of our churches mutual friends and vendors is that you guys got direct fiber hookups between campuses and are using that to send video/networking/data between campuses.
Is that correct?
If it isn’t and you guys are doing leased lines or something then your latency is probably reasonably close and you can disregard the rest of this message.
However if you do have private lines, as I believe you do, then your latency to either of the two campuses on the data layer should be between 4 to 8 ms _round trip_.
Furthermore, at the data layer, your latency is in the switches, interfaces, and routers; not so much the physical distance (light is so fast it can make it round trip from/to any of your campuses in less than 1ms). As such there shouldn’t be a considerable difference in latency no matter what the campus.
The latencies you are stating would then seem to me to be above the data layer, a combination of latency in the transport system (HDSDI RX/TX Transcoders, yes?) and maybe multitude of reclocking stages on the AES lines?
I gather you are using audio over video so that you can send the audio with the video so the remote video servers can time delay the message and have in sync audio, correct?
Do you know how much latency the servers/transcoders are adding?
Also, have you looked hard at all the timing. While your london will grab time from your console via AES, is your console in sync with video land, where I ASSume the transcoder is?
In all likelihood the transcoders are probably genlocked to the video system. If your console is not similarly clocked to video, you will have a few MS of clocking delays there.
I believe you guys are using evertz master clock(s) in video land. IIRC they have timed wordclock spigots on them. You should try timing your console to video land, your london will fall in time with the console, and the london and transcoder should be in time, albeit they are getting there sync from different places.
The same thing would have to be done on the other ends, with the exception that they probably don’t have londons, thus the console would only need to be in time with their respective video lands.
Alternatively, have you considered running a parallel system for real time audio? Go ahead and let the message audio flow through the transcoders, but use a dedicated IP audio system for these other “creative” uses. There are audio over IP systems that have sub 1ms end to end latency. Add in network latency you have <7ms one way trips. If you go further and manage your latency very carefully (which may even involve using some analog cards instead of digital to help with clocking issues) you should be able to do sub 10 ms round trips, which is fast enough to really do something.
After re-reading this, I apologize for spamming your board, I probably should have just called you. Oh well.
Or maybe you already thought about all this and didn’t want to bore us all with it…… 🙂
Karl P
We know that any latency we were encountering was a result of gear at campuses. Audio is transmitted embedded with the video signals. I might be incorrect, but I’m pretty sure that they make sure everything is in sync before it is transmitted. So there’s time there. I’m sure there’s also time added from sample rate converters built into all the digital gear. I guess the bottom line is there are probably some things we could do if we really wanted to, to reduce the latency a bit more. But at the end of the day, this isn’t something we are doing very often so justifying the time and effort into reducing transmission latency is pretty hard right now. The main purpose behind implementing the system was for message transmission, and our ability to pull something like this off is really just a nice addition. That’s not to say that at some point we might want to investigate possibilities, but it’s not a high priority at this time.
We have talked about dedicated audio lines, but once again it’s just not a need for us right now. I think it would be very cool if we had every input from every campus showing up on a matrix at every campus someday with our audio guys able to grab any input they want, but that would be overkill for what we’re doing now and at a minimum for the next 3-5 years. I think long before we get to that point we would be better served by setting up equivalent loudspeaker systems in our rooms tuned identically so that a mix from one location has a much higher translation rate at any campus. We’ve already seen some of the potential for this on my campus where we have two non-identical systems in our main rooms. When those rooms are close, things go back and forth pretty well. I think the next step on the larger scale is really to update our 13 year old systems at my campus to be on par with the modern loudspeaker systems at our other campuses. Then it’s just a matter of standardizing a tuning curve which I think we could all settle on if we were doing a lot of these types of things.
I certainly won’t disagree with the need of balancing out your systems and tuning curves. That would certainly seem to be a big benefit to what you are doing and most certainly a higher priority.
Also there is no doubt that what you have is cool, and most likely totally appropriate for your current usage.
However it would seem like there is much more time than money involved in the Wide Area Audio Network than in your other “priority” projects which seem to be largely money intensive at the moment.
So, if you ever find yourself sitting around on a Monday morning wondering what to do…… 🙂
Karl P