---BREAKAWAY CIVILIZATION ---ALTERNATIVE HISTORY---NEW BUSINESS MODELS--- ROCK & ROLL 'S STRANGE BEGINNINGS---SERIAL KILLERS---YEA AND THAT BAD WORD "CONSPIRACY"--- AMERICANS DON'T EXPLORE ANYTHING ANYMORE.WE JUST CONSUME AND DIE.---
49ers CTO Kunal Malik (left) and Senior IT director Dan Williams (right) stand in front of Santa Clara Stadium.
Jon Brodkin
When the San Francisco 49ers' new stadium opens for the 2014 NFL
season, it is quite likely to have the best publicly accessible Wi-Fi
network a sports facility in this country has ever known.
The 49ers are defending NFC champions, so 68,500 fans will
inevitably walk into the stadium for each game. And every single one of
them will be able to connect to the wireless network, simultaneously,
without any limits on uploads or downloads. Smartphones and tablets
will run into the limits of their own hardware long before they hit the
limits of the 49ers' wireless network.
Enlarge/ A model of Santa Clara Stadium, with a wall painting visible in the background.
Until now, stadium executives have said it's pretty much impossible
to build a network that lets every single fan connect at once. They've blamed this
on limits in the amount of spectrum available to Wi-Fi, despite their
big budgets and the extremely sophisticated networking equipment that
largesse allows them to purchase. Even if you build the network
perfectly, it would choke if every fan tried to get on at once—at least
according to conventional wisdom.
But the people building the 49ers' wireless network do not have
conventional sports technology backgrounds. Senior IT Director Dan
Williams and team CTO Kunal Malik hail from Facebook, where they spent
five years building one of the world's largest and most
efficient networks for the website. The same sensibilities that power
large Internet businesses and content providers permeate Williams' and
Malik's plan for Santa Clara Stadium, the 49ers' nearly half-finished
new home.
"We see the stadium as a large data center," Williams told me when I visited the team's new digs in Santa Clara.
I had previously interviewed Williams and Malik over the phone, and
that's when they first told me they planned to make Wi-Fi so ubiquitous
throughout the stadium that everyone could get on at once. I had never
heard of such an ambitious plan before—how could this be possible?
Today’s networks are impressive—but not unlimited
An expansive Wi-Fi network at this year's Super Bowl
in the New Orleans Superdome was installed to allow as many as 30,000
fans to get online at once. This offloaded traffic from congested
cellular networks and gave fans the ability to view streaming video or
do other bandwidth-intensive tasks meant to enhance the in-game
experience. (Don't scoff—as we've noted before, three-plus-hour NFL
games contain only 11 minutes of actual game action,
or a bit more if you include the time quarterbacks spend shouting
directions at teammates at the line of scrimmage. There is plenty of
time to fill up.)
Superdome officials felt a network allowing 30,000 simultaneous
connections would be just fine, given that the previous year's Super
Bowl saw only 8,260 at its peak. They were generally right, as the
network performed well, even for part of the game's power outage.
The New England Patriots installed a full-stadium Wi-Fi network this
past season as well. It was never used by more than 10,000 or so people
simultaneously, or by more than 16,000 people over the course of a full
game. "Can 70,000 people get on the network at once? The answer to that
is no," said John Brams, director of hospitality and venues at the
Patriots' network vendor, Enterasys. "If everyone tried to do it all at
once, that's probably not going to happen."
But as more fans bring smart devices into stadiums, activities like
viewing instant replays or live camera angles available only to ticket
holders will become increasingly common. It'll put more people on the
network at once and require bigger wireless pipes. So if Williams and
Malik have their way, every single 49ers ticket holder will enjoy a
wireless connection faster than any wide receiver sprinting toward the
end zone.
"Is it really possible to give Wi-Fi to 68,500 fans at once?" I
asked. I expected some hemming and hawing about how the 49ers will do
their best and that not everyone will ever try to use the network at
once anyway.
"Yes. We can support all 68,500," Williams said emphatically.
How?
"How not?" he answered.
Won't you have to limit the capacity each fan can get?
Again, absolutely not. "Within the stadium itself, there will
probably be a terabit of capacity. The 68,500 will not be able to
penetrate that. Our intentions in terms of Wi-Fi are to be able to
provide a similar experience that you would receive with LTE services,
which today is anywhere from 20 to 40 megabits per second, per user.
"The goal is to provide you with enough bandwidth that you would
saturate your device before you saturate the network," Williams said.
"That's what we expect to do."
Fans won't be limited by what section they're in, either. If the
49ers offer an app that allows fans to order food from their seats, or
if they offer a live video streaming app, they'll be available to all
fans.
"The mobile experience should not be limited to, 'Hey, because you
sit in a club seat you can see a replay, but because you don't sit in a
club seat you can't see a replay,'" Malik said. "That's not our
philosophy. Our philosophy is to provide enhancement of the game
experience to every fan." (The one exception would be mobile features
designed specifically for physical features of luxury boxes or club
seats that aren't available elsewhere in the stadium.)
It’s the design that counts
Current stadium Wi-Fi designs, even with hundreds of wireless access
points distributed throughout a stadium, often can support only a
quarter to a half of fans at once. They also often limit bandwidth for
each user to prevent network slowdowns.
The Patriots offer fans a live video and instant replay app, with
enough bandwidth to access video streams, upload photos to social
networks, and use the Internet in general. Enterasys confirmed to Ars
that the Patriots do enforce a bandwidth cap to prevent individual users
from overloading the network, but Enterasys would not say exactly how
big the cap is. The network has generally been a success, but some users
of the Patriots app have taken to the Android app store to complain about the stadium Wi-Fi's performance.
According to Williams, most current stadium networks are limited by a
fundamental problem: sub-optimal location of wireless access points.
"A typical layout is overhead, one [access point] in front of the
section, one behind the section, and they point towards each other," he
said. "This overhead design is widely used and provides enough coverage
for those using the design."
Williams would not reveal the exact layout of the 49ers' design,
perhaps to prevent the competition from catching on. How many access
points will there be? "Zero to 1,500," he said in a good-natured attempt
to be both informative and vague.
That potentially doubles or quadruples the typical amount of stadium
access points—the Super Bowl had 700 and the Patriots have 375. But this
number isn't the most important thing. "The number of access points
will not give you any hint on whether the Wi-Fi is going to be great or
not," Malik said. "Other factors control that."
If the plan is to generate more signal strength, just adding more
access points to the back and front of a section won't do that.
The Santa Clara Stadium design "will be unique to football stadiums,"
Williams said. "The access points will be "spread and distributed. It's
really the best way to put it. Having your antennas distributed evenly
around fans." The 49ers are testing designs in Candlestick Park and
experimenting with different access points in a lab. The movement of
fans and the impact of weather on Wi-Fi performance are among the
factors under analysis.
Enlarge/ This balcony section will someday be surrounded by wireless access points.
"Think of a stadium where it's an open bowl, its raining, people are
yelling, standing, how do you replicate that in your testing to show
that if people are jumping from their seats, how is Wi-Fi going to
behave, what will happen to the mobile app?" Malik said. "There is a lot
that goes on during a game that is hard to replicate in your conceptual
simulation testing. That is one of the big challenges where we have to
be very careful."
"We will make great use of Candlestick over the next year as we
continue to test," Williams said. "We're evaluating placement of APs and
how that impacts RF absorption during the game with folks in their
seats, with folks out of their seats."
Wi-Fi will be available in the stands, in the suites, in the
walkways, in the whole stadium. The team has not yet decided whether to
make Wi-Fi available in outdoor areas such as concourses and parking
lots.
The same could theoretically be done at the 53-year-old Candlestick
Park, even though it was designed decades before Wi-Fi was invented.
Although the stadium serves as a staging ground for some of the 49ers'
wireless network tests, public access is mainly limited to premium
seating areas and the press box.
The reason Wi-Fi in Candlestick hasn't been expanded is a practical
one. With only one year left in the facility, the franchise has decided
not to invest any more money in its network. But Williams said 100
percent Wi-Fi coverage with no bandwidth caps could be done in any type
of stadium, no matter how old. He says the "spectrum shortage" in
stadiums is just a myth.
With the new stadium still undergoing construction, it was too early
for me to test anything resembling Santa Clara Stadium's planned Wi-Fi
network. For what it's worth, I was able to connect to the 49ers' guest
Wi-Fi in their offices with no password, and no problems.
The 2.4GHz problem
There is one factor preventing better stadium Wi-Fi that even the
49ers may not be able to solve, however. Wi-Fi works on both the 2.4GHz
and 5GHz bands. Generally, 5GHz is better because it offers more
powerful signals, less crowded airwaves and more non-overlapping
channels that can be devoted to Wi-Fi use.
The 2.4GHz band has 11 channels overall and only three that don't
overlap with each other. By using somewhat unconventionally small 20MHz
channels in the 5GHz range, the 49ers will be able to use about eight
non-overlapping channels. That's despite building an outdoor stadium,
which is more restricted than indoor stadiums due to federal
requirements meant to prevent interference with systems like radar.
Each 49ers access point will be configured to offer service on one
channel, and access points that are right next to each other would use
different channels to prevent interference. So even if you're
surrounding fans with access points, as the 49ers plan to, they won't
interfere with each other.
But what if most users' devices are only capable of connecting to the
limited and crowded 2.4GHz band? Enterasys said 80 percent of Patriots
fans connecting to Wi-Fi this past season did so from devices supporting
only the 2.4GHz band, and not the 5GHz one.
"You have to solve 2.4 right now to have a successful high-density public Wi-Fi," Brams said.
The iPhone 5 and newer Android phones and tablets do support both the
2.4GHz and 5GHz bands, however. Williams said by the time Santa Clara
Stadium opens in 2014, he expects 5GHz-capable devices to be in much
wider use.
When asked if the 49ers would be able to support 100 percent of fans
if most of them can only connect to 2.4GHz, Williams showed a little
less bravado.
"For those 2.4 users we will certainly design it so that there's less
interference," he said. "It is a more dense environment if you are
strictly constrained in 2.4, but we are not constrained in 2.4. We're
not trying to answer the 2.4 problem, because we have 5 available."
"It's 2013, we have another year and a half of iteration," he also
said. "We'll probably be on, what, the iPhone 7 by then? The move to
5GHz really just makes us lucky. We're doing this at the right time."
Building a stadium in Facebook’s image
Williams and Malik both joined the 49ers last May. Malik was hired
first, and then brought his old Facebook friend, Williams, on board.
Malik had been the head of IT at Facebook, while Williams was the
website's first network engineer and later a director. They both left
the site, basically because they felt there was nothing left to
accomplish. Williams did some consulting, and Malik initially planned to
take some time off.
Williams was a long-time 49ers season ticket holder, but that was far from the only thing that sold him on coming to the NFL.
"I had been looking for something challenging and fun again,"
Williams said. "Once you go through an experience like Facebook, it's
really hard to find something that's similar. When Kunal came to me, I
remember it like it was yesterday. He said, 'If you're looking for
something like Facebook you're not going to find it. Here's a
challenge.'"
"This is an opportunity to change the way the world consumes live
sports in a stadium," Malik said. "The technology problems live sports
has today are unsolved and no one has ever done what we are attempting
to do here. That's what gets me out of bed every day."
Williams and Malik have built the 49ers' network in Facebook's image.
That means each service—Wi-Fi, point-of-sale, IPTV, etc.—gets its own
autonomous domain, a different physical switching system to provide it
bandwidth. That way, problems or slowdowns in one service do not affect
another one.
"It's tribal knowledge that's only developed within large content
providers, your Facebooks, your Googles, your Microsofts," Williams
said. "You'll see the likes of these large content providers build a
different network that is based on building blocks, where you can scale
vertically as well as horizontally with open protocols and not
proprietary protocols.
"This design philosophy is common within the content provider space
but has yet to be applied to stadiums or venues. We are taking a design
we have used in the past, and we are applying it here, which makes sense
because there is a ton of content. I would say stadium networks are 10
years behind. It's fun for us to be able to apply what we learned [at
Facebook]."
The 49ers are still evaluating what Wi-Fi equipment they will use.
The products available today would suit them fine, but by late 2014
there will likely be stadium-class access points capable of using the
brand-new 802.11ac protocol,
which allows greater throughput in the 5GHz range than the widely used
802.11n. 11ac consumer devices are rare today, but the 49ers will use
802.11ac access points to future-proof the stadium if appropriate gear
is available. 11ac is backwards compatible with 11n, so supporting the
new protocol doesn't leave anyone out—the 49ers also plan to support
previous standards such as 11a, 11b, and 11g.
802.11ac won't really become crucial until 802.11n's 5GHz
capabilities are exhausted, said Daren Dulac, director of business
development and technology alliances at Enterasys.
"Once we get into 5GHz, there's so much more capacity there that 11ac
doesn't even become relevant until we've reached capacity in the 5GHz
range," he said. "We really think planning for growth right now in 5GHz
is acceptable practice for the next couple of years."
Santa Clara Stadium network construction is expected to begin in
Q1 2014. Many miles of cabling will support the "zero to 1,500" access
points, which connect back to 48 server closets or mini-data centers in
the stadium that in turn tie back to the main data center.
"Based on service type you plug into your specific switch," Williams
said. "If you're IPTV, you're in an IPTV switch, if you're Wi-Fi you're
in a Wi-Fi switch. If you're in POS [point-of-sale], you're in a POS
switch. It will come down to a Wi-Fi cluster, an IPTV cluster, a POS
cluster, all autonomous domains that are then aggregated by a very large
fabric, that allows them to communicate lots of bandwidth throughput,
and allows them to communicate to the Internet."
Whereas Candlestick Park's network uses Layer 2 bridging—with all of
the Wi-Fi nodes essentially on a single LAN— Santa Clara Stadium will
rely on Layer 3 IP routing, turning the stadium itself into an
Internet-like network. "We will be Layer 3 driven, which means we do not
have the issue of bridge loops, spanning tree problems, etc.," Williams said.
Keeping the network running smoothly
Wireless networks should be closely watched during games to identify
interference from any unauthorized devices and identify usage trends
that might result in changes to access points. At the Patriots' Gillette
Stadium, management tools show bandwidth usage, the number of fans
connected to each access point, and even what types of devices they're
using (iPhone, Android, etc.) If an access point was overloaded by fans,
network managers would get an alert. Altering radio power, changing
antenna tilt, or adding radios may be required, but generally any major
changes are made between games.
Enlarge/ Dashboard view of Patriots' in-game connectivity.
Enterasys
"In terms of real-time correction, it depends on what the event is,"
said John Burke, a senior architect at Enterasys. "Realistically, some
of these APs are overhead. If an access point legitimately went down and
it's on the catwalk above 300 [the balcony sections] you're not going
to fix that in the game. That's something that would have to wait."
So far, the Patriots' capacity has been enough. Fans have yet to
overwhelm a single access point. Even if they did, there is some overlap
among access points, allowing fans to get on in case one AP is
overloaded (or just broken).
The 49ers will use similar management tools to watch network usage
and adjust access point settings in real time during games. "We expect
to overbuild and actually play with things throughout," Williams said.
"Though we are building the environment to support 100 percent capacity,
we do not expect 100 percent capacity to be used, so we believe we will
be able to move resources around as needed [during each game].”
The same sorts of security protections in place in New England will
be used in Santa Clara. Business systems will be password-protected and
encrypted, and there will be encrypted tunnels between access points and
the back-end network. While that level of protection won't extend to
the public network, fans shouldn't be able to attack each other, because
peer-to-peer connections will not be allowed.
What if the worst happens and the power goes out? During the Super
Bowl's infamous power outage, Wi-Fi did stay on for at least a while.
Williams and Malik acknowledged that no system is perfect, but they said
that they plan for Wi-Fi uptime even if power is lost.
"We have generators in place, and we'll have UPS systems, so from a
communications standpoint our plan is to keep all the communication
infrastructure up and online [during outages]," Williams said. "But all
of this stuff is man-made."
A small team that does it all
Believe it or not, the 49ers have a tech team of less than 10 people,
yet the organization is designing and building everything itself.
Sports teams often outsource network building to carriers or equipment
vendors, but not the 49ers. Besides building its own Wi-Fi network, the
team will build a carrier-neutral distributed antenna system to boost
cellular signals within the stadium.
Enlarge/ Williams, a longtime season ticket holder, keeps his 49ers jacket at his desk.
Jon Brodkin
"We are control freaks," Williams said with a laugh. He explained
that doing everything themselves makes it easier to track down problems,
accept responsibility, and fix things. They also feel the need to take
ownership of the project because none of the existing networks in the
rest of the league approach what they want to achieve. There is a lot of
low-hanging fruit just from solving the easy problems other franchises
haven't addressed, they think.
Not all the hardware must be in-house, though. The 49ers will use
cloud services like Amazon's Elastic Compute Cloud when it makes sense.
"Let's say we want to integrate a POS system with ordering," Malik
said. "If you have an app that lets you order food, and there's a point
of sale system, all the APIs and integration need to sit in the cloud.
There's no reason for it to sit in our data center."
There are cases where the cloud is clearly not appropriate, though.
Say the team captures video on site and distributes it to fans'
devices—pushing that video to a faraway cloud data center in the middle
of that process would slow things down dramatically. And ultimately, the
49ers have a greater vision than just providing Wi-Fi to fans.
When I toured a preview center meant to show off the stadium
experience to potential ticket buyers, a mockup luxury suite had an iPad
embedded in the wall with a custom application for controlling a
projector. That provides a hint of what the 49ers might provide.
"Our view is whatever you have at home you should have in your
suite," Williams said. "If that means there's an iPad on the wall or an
application you can use, hopefully that's available. Your life should be
much easier in this stadium."
And whatever applications are built should be cross-platform. As
Malik said, the 49ers are moving away from proprietary technologies to
standards-based systems so they can provide nifty mobile features to
fans regardless of what device they use.
Williams and Malik are already working long hours, and their jobs
will get even more time-intensive when network construction actually
begins. But they wouldn't have it any other way—particularly the
longtime season ticket holder Williams.
When work is "tied to something that you love deeply, which is
sports, and tied to your favorite team in the world, that's awesome,"
Williams said. "I'm crazy about it, man. I get super passionate."
Posts
No comments:
Post a Comment