Pandemic Downtime Let Kraft Rebuild The IT Behind Patriots Football

In a lot of ways, the impact of the COVID-19 pandemic on The Kraft Group was no different than most other companies around the world. The organization – a holding company best known for its ownership of the New England Patriots football team and its home field Gillette Stadium – had to send the bulk of its workers home to work in hopes of slowing the spread of the deadly virus.

That meant rapidly building the capabilities to enable those remote workers to actually work remotely, including buying and distributing 1,200 Dell laptops to employees. However, it also presented the organization’s IT staff the opportunity to accelerate an already planned refresh of the datacenter facilities housed in the stadium in Foxborough, Massachusetts, but which also serve all of its other diverse businesses and support its 9,500 employees and hundreds of customers around the world.

“As a whole, because of our business, because of our manufacturing locations, and being a [COVID] vaccination site, we operated on-site throughout the pandemic,” Kraft Group CIO Michael Israel tells The Next Platform. “From an IT, legal, HR perspective, for our core businesses, the support teams were here throughout the pandemic. My team was never remote. Because we had no real events going on in the stadium – no concerts; we had football and soccer, but not with crowds – for periods of time, it gave us latitude in some projects that under normal circumstances, we’d be working around what’s happening around us.”

That includes the Patriots games that run through the fall and into the winter, which limit how much the IT folks can do during those months. Even with Tom Brady no longer in the building – and now out of football altogether, with his (second) retirement this week – Gillette is packed with fans. However, “during COVID, because I didn’t have fans coming in here, we had some unique opportunities and we used that to our advantage,” Israel says.

That advantage included setting up a second datacenter at the stadium populated by three Dell VxRail hyperconverged infrastructure systems and PowerStore block and file storage appliances for both datacenter and edge workloads, helping to reduce The Kraft Group’s overall storage footprint by 40 percent. The company uses Dell’s PowerScale NAS systems across its various business units, including storing 15,000 tapes from 25 years of broadcasting from its sports and production operations.

Israel and his team also use PowerScale for internal files, daily backups, and disaster recovery purposes, he says.

The Kraft Group now operates two datacenters in the stadium, with the new one housing the VxRails and on-prem PowerStore systems. The datacenters hold the IT not only for Gillette Stadium, the Patriots, and New England Revolution pro soccer team (which along with Patriot Place shopping area around the stadium and Kraft Analytics Group make up the organization’s Sports + Entertainment company), but also The Kraft Group’s other operating businesses, such as Rand-Whitney Container, Rand-Whitney Containerboard, Rand-Whitney Recycling, International Forest Products, and New-Indy Container Corp, with eight manufacturing plants in the region.

Other sites also have systems like servers in place to support manufacturing operations, but in terms of IT, it’s all run out of Gillette.

The older datacenter still holds such infrastructure as the older PowerEdge systems and there is a colocation facility about 75 miles away that houses a Vblock environment that includes on-prem compute and storage for disaster recovery.

“Before building the new datacenter, we were building on top of old infrastructure,” Israel says. “By stripping everything out, we now understand, one, how everything was connected together, so we re-documented the entire physical network itself. We had one primary datacenter. We now have two datacenters in which with our core network, we have redundant paths coming into the stadium. Network cores also split between two locations and we have two different paths through. If I have an IDF that goes offline or a switch in an IDF, we no longer have one direct path in one datacenter. We have two paths, so in a scenario where a fiber run gets cut, something gets cut, we have two ways to get back. We have two distinct paths out to the Internet, so if we were to lose connectivity on one or in our back piece, we go to separate local exchange carriers, we go to separate paths from our telecom providers. It’s completely divergent.”

Israel’s team is rebuilding the old datacenter, but the key is the split redundancy that comes with having both, he says. If half the network core goes down, they are still connected elsewhere.

“At this point, with the exception of one use case, all of our servers are consolidated into VxRail platforms and everything’s virtualized,” the CIO says. “VxRails are basically running our entire business. The only use case that’s outside of the VxRails is we have an array of PowerEdge servers that are running our physical security cameras. Those are now stood up in the new datacenter.”

The VxRails also helped The Kraft Group save on datacenter space. For example, having workloads associated with the Patriots sitting on a single four-node VxRail saves about two-thirds the space of a typical cabinet. At the same time, having a single set of virtual machine clusters makes management easier. “Where I would have had two full cabinets full of individual servers, space-savings … and the overall physical items we have to manage is far less,” he says.

Overall, the Kraft Group has reduced its datacenter footprint by 20,000 square feet.

Other efforts during this time included installing 110 new intermediate distribution frames (IDFs) that offer new and redundant fiber paths to both the old and new datacenters in the stadium and which replaced legacy IDFs that were 20 or more years old, enabling the company to rip out its legacy cabling. The IT team also rolled out new Wi-Fi 6 wireless connectivity, consolidating all the telecommunications lines into the new datacenter and eliminating 90 percent of the telco lines, Israel says.

The company also is building up what it does at the edge, from increasing security via cameras to improving the mobile experience for fans, and using AI to accelerate the analysis of the all the data that’s being collected. All this is important not just for the fan experience – which in 2026 will include hosting World Cup soccer games – but also to ensure good operations for such use cases as video collaboration for workers who are at the stadium or remote.

The break the pandemic gave Israel and his IT group enabled them to accomplish in one year what otherwise would have taken three, he says, adding that “this stadium is used full time all year long, even though it may not be football season. In February, March and April, soccer starts. We have venue events going on almost every day and that has increased coming out of the pandemic. There’s a lot of pent-up demand. There’s never a good time to take sections of the building offline.”

The organization is about 99 percent done with its server and storage work. Now they are looking at what’s coming up. That includes evaluating whether to do further datacenter consolidation, such as moving from three VxRail systems to one. They are weighing the risks involved with going with a single platform and the possible cost savings.

“Really it becomes more of an economic discussion,” Israel says. “What’s my cost from a TCO perspective of managing three systems vs. just managing one? Is there more to be gained from a simplification perspective, bringing things further down? We’re going through an exercise right now ahead of when our next lifecycle changes are. What does tomorrow bring us? Since we’ve done our initial installs, some of the groups have broken out into more cloud scenarios, so now we’re looking at where our data is sitting that’s outside [of the datacenter].”

They’re also looking ahead to later this year, when Microsoft stops supporting Windows Server 2012 and 2012 R2. The planning already has begun, determining the number of instances of the operating system they have, and how to schedule the server upgrades during the year.

That said, the biggest project for 2023 is around data governance, including how old the data is, how it’s classified, and how it’s stored and backed up. With greater data governance, simply taking daily snapshots and weekly backups won’t work. Financial data needs to be treated differently than email.

“We’re looking at, how do we change our datasets, how we change how we back up Windows so that if I want email data broken out, if it expires here, it needs to expire in the back-up sense as well,” he says. “That’s something that we are still working on. One component of it is, here’s our governance window. The other component is end users who don’t necessarily want to give up data. They haven’t looked at it in ten years but there’s a sense of security knowing we still have it. We’re working through that.”

Sign up to our Newsletter

Featuring highlights, analysis, and stories from the week directly from us to your inbox with nothing in between.
Subscribe now

Be the first to comment

Leave a Reply

Your email address will not be published.


*


This site uses Akismet to reduce spam. Learn how your comment data is processed.