Echo Lake Foods Disaster
Other coverage of this event:
XENITEL CONQUERS MONUMENTAL TASK
January 30th 2013, the main plant and headquarters of Echo Lake Foods – a large producer of frozen breakfast food creations with over 800 employees and $150 million annual sales, was severely damaged by what is now known to be the second largest fire in Wisconsin history. Over eighty fire companies with 300+ firefighters responded. After sixteen hours and thankfully no injuries – the smoke cleared, but Echo Lake Foods was out of business. All of their IT systems were destroyed impacting both existing operations as well as a total loss of all historical data in the profitable business they had been operating for over 70 years. Included in the piles of twisted steel, rubble and debris were computer servers, networking equipment and lost data. There were two remaining plants not hit by the fire but realistically they were critically off line with the main base of operations down unable to operate solo. Echo Lake Foods was able to turn to Xenitel. Prior to 2013 they had collaborated to build a disaster recovery solution that was ready to roll into place from their cloud.
“Our on-site monitors started to indicate a major rise in heat” says Brian Quandt, IT Director at Xenitel, a Waukesha WI based IT service provider. “We knew something had gone terribly wrong so we made some calls and remotely started an orderly shutdown of all IT systems”. Echo Lake Foods was losing money by the minute but Xenitel was able to jump to action just as fast.
“We had backups of all of our ERP systems and data from our nightly vaulting services” Says Brad Temby, Systems Engineer with Xenitel. Vaulting allowed Echo’s data to be transmitted to off-site back up servers during non–peak hours. However, “Having backup data is only half the battle in restoration” says Temby, “You need a platform to restore it TO is the real issue”. This is where a cloud based disaster recovery solution comes in handy!
Lee Fischer, CFO of Elkin, Echo Lake Food’s parent company, describes the disaster recovery planning process. “When you review business continuity from a systems perspective, you have to ask, what good is a back up of the data if you have no where to put that data.” “It has to be a whole plan, with many options and cost models to consider”.
Utilizing their DR cloud, Xenitel was able to rebuild their complex server environment. They leveraged their clouds virtual servers to restore terabytes of Echo Lake’s data immediately. While data was being restored, the job of connecting several make-shift satellite locations began. Critical business functions such as accounting, human resources, and customer service all required connections to the newly created virtual servers. Day to day operations such as the 800 employees relying upon Echo for payroll had to completed within the week.
Michael Herrick, Director of Cisco networking at Xenitel, re-routed key customer network circuits to the Xenitel data center. This allowed Echo to use freestanding structures survived by the fire as a new headquarters for the company. Being treated like family while their company showed concern over payroll and safety the employees pitched in quickly and a few basements became “satellite” operations during this emergency. Xenitel used simple VPN (virtual private network) to ensure all temporary operations had the proper security and access and did not require any new equipment not commonly found in any home today between cable and internet.
Additionally, Echo had chosen to off load mission critical but simple applications like messaging and email to Xenitel’s cloud which meant those systems never went down in the fire as they were already off site in Xenitel’s cloud. “Echo chose to host most of their own systems” says Herrick, “however all of Echo’s messaging and email was hosted in our cloud”. This allowed Echo Lake to maintain critical links with customers and suppliers without interruption.
Critical to the restoration were the two remaining plants. Warehousing and shipping functions were able to pick back up sooner than later to continue to produce and ship products. “Looking at these (disaster recovery) plans on paper, its easy to have doubts in the back of your mind” says Lou Morelli , Director of Emerald ERP , Echo’s main warehousing and supply chain software. “Even with the complexity of this disaster, the Xenitel team executed their plan flawlessly getting our systems back online in no time. ”
According to a recent survey undertaken by freelancing site PeoplePerHour.com of among 1,300 small to medium business owners it was found that as many as 74% do not use ‘Cloud’ computing. The planning for disaster recovery with Echo Foods has only taken about ninety days from concept to completion. ”We looked at many designs and than selected a plan that best fit the company’s business plan and budget.” says Fischer.
Only three weeks later, in a section of the plant spared by the fire, Echo Lake Foods began producing breakfast foods once again. Plans for rebuilding the tattered plant are now in full swing and the other plants are operating normally. Happy to finally get a nights rest, the Xenitel team built a new server platform and transferred systems back to Echo.
Jerry Warntjes, the General Manager of Echo Lake Foods summed up the situation in one word, “Amazing! We had no idea what to expect, luckily our IT partner had all the details figured out”
Echo Lake Foods:
Echo Lake Foods has been providing quality products and services to many large food service organizations in the U.S. for over 70 years, specializing in the production of frozen breakfast products/components including frozen liquid egg, egg patties/scrambled, frittata’s, French toast, grilled toast, pancakes, waffles, and other frozen meal creations. Echo Lake has become a leader in retail co-pack delivering great tasting products with outstanding value while meeting significant capacity requirements. The company works with many segments of foodservice including large food manufacturers, QSR’s, food distributors, retail marketers, sandwich makers, Government entities, large school districts, pizza manufacturers, assemblers and grocery/specialty retailers.
Xenitel is a premiere provider of IT outsourcing and consulting services, specializing in Disaster Recovery design/implementation and overall systems organization and availability planning. Headquartered outside Milwaukee, Xenitel has become the full-service IT department of choice for small and medium-sized businesses needing unique remote technology capabilities to support offices in remote locations. Specialties include IT infrastructure and operations management services, application hosting, security assessment, systems integration, backup & recovery and IT consulting, as well as the procurement of hardware and software for your business.