I’m writing to apologize for the disruption and inconvenience many of you experienced between Thursday afternoon, October 19th, and Saturday morning, October 20th. Our website went offline and we know that the outage affected many of you. We’ve had a lot of questions about what happened, so I thought I’d take a few minutes to answer some of those questions.
On behalf of our whole team, I’d like to thank all of you who called, emailed, and talked to us on Facebook for your support and kind words. It meant a lot to hear from you during a difficult time.
First, we were not hacked. Your credit card and personal information aren’t compromised.We do not keep credit card information on our servers. None of your addresses or other contact information was compromised, and your order history, awards points, and invoices are right where they should be. By Saturday morning, our website was back up and fully functional.
What Happened?
Our hosting provider, a data-center company called OpusInteractive, with four data centers around the country (including two in the greater Portland area,) had a hardware and software failure that took a large number of businesses offline; ours among them. OpusInteractive was unable to solve the problem for more than 30 hours. When their service was reestablished they were unable to restore our website. It was a massive failure on their part, one that is inexcusable, and one that had pretty tough repercussions for our customers and our business. We eventually had a team of engineers work all night Friday to completely rebuild and test our website and get us running by Saturday morning.
The key thing for data-centers is uptime and availability; it’s the only service they offer. Essential has a very experienced tech team inside the company and relies on some very experienced partners. To do what we do, and do it successfully, we use a lot of Enterprise level software and hardware. When it comes to the website we don’t cut any corners; while we have a rack of servers in-house supporting our internal accounting systems, formulation, and manufacturing software, and myriad utilities that allow us to operate. As with many companies our size, our website is hosted offsite at a data center that guarantees 99.98% uptime 365 days a year. The data center provides the redundant systems, virtual servers, and access to the fiber optic data ring that keeps websites up and running every minute of every day.
Going forward we will be changing our hosting partner (duh) and have already set up a redundant system off-site and a mirrored server onsite at our main facility to serve as an emergency backup if we run into this sort of problem again.
All businesses are challenging, and each has their own keys to success. First and foremost, our customers are the keys to our success. At Essential Wholesale & Labs, we basically do two things: we work very hard to create and manufacture great natural and organic cosmetic products and we serve you, our customers around the world through our website.
I hope this helps you understand the outage. One of our values is to be transparent and open with our customers. If there are any other questions we can answer about this please let us know.
Thank you for your continued business and support.
John Landforce, President & CEO
