EI Outage – Lessons Learnt

[bigmango]

What lessons do think EI should have learnt ?

I have read the Incident Report – two comments

- The time taken to initiate their Disaster Recovery Plan needs to reviewed

- The creation of a holding page (based on a snapshot of the advertisers details taken daily or every few hours)
needs to be built in to the normal operation of the site so that in the event of an outage the holding page is automatically available

What lessons did you learn yourself ?

[SteveB]

Any compensation deal hammered out yet

You trying to be funny?

[SteveB]

It is impossible for us to know what happened. The “incident report” says that it was “external service provider” (yeah, blame someone else but the twitter report blamed “routine maintenance”. Maybe it was “routine maintenance” carried out by an “external service provider”?

In any case it matters little to us or the escorts what the problem was. What does matter is the likelihood of it happening again, and to the escorts, what compensation they are going to get. I suspect escorts will not get anything for lost business but may get some pittance or credit for ads not appearing etc.

The site was down for about eleven hours. Even so, that would probably not be considered a “major outage” by some IT standards. I heard a comment some time ago that if a company suffers a major IT outage, especially if there is a significant data loss, there is a fifty percent chance that company will not survive.

Other companies, major banks, for example, have had worse and longer-lasting incidents. EI did respond fairly quickly, kept everyone informed via twitter, had a disaster plan and did get the system up and running without any data loss.

However, an escort who paid good money for an expensive hotel room or apartment and then didn’t get any business will not be impressed.

[Carolyne]

Yeah ,I followed Twitter when the site was down ,and they told its a maintenance by the site ,strange .Another strange with the time ,in the report they said around 3:30 they could displayed all the numbers so we could get phone calls ,but on the Twitter was much later that .It was 17:13 ,when they displayed the numbers.

[bigmango]

YES - I agree - I followed the Twitter account & it was after 17:00 when the page of advertisers details appeared.

[Stephanie]

YES - I agree - I followed the Twitter account & it was after 17:00 when the page of advertisers details appeared.

Perhaps it's time to sign up with Twitter .. ---- then again , maybe not .

[SteveB]

Perhaps it's time to sign up with Twitter .. ---- then again , maybe not .

No need. Can see EI twitter without signing up if all you want is just to see the tweets.

https://twitter.com/escortireland

[SteveB]

Bad enough here without being on Twitter.

[DevTeam]

Hi Everyone,

We have been dealing direct with advertisers regarding compensation. This is not something up for public discussion.

The management staff from each of our offices met yesterday and today to review and review various policies. We learned a lot of lessons from this.

We already have full HA on all of our equipment with N+1 on every resource, there are however a few points of failure that we had not envisaged ever failing (due to the size of the corporations offering the services who claim to have N+2 on their equipment) on critical parts of our infrastructure that we now know is not the case.

We hope that we never have a similar situation again and i commend the efforts of our whole team during the incident.

Thanks

[SteveB]

Send for First Engineer Montgomery Christopher Jorgensen Scott. He'll sort it out!



Personally, I think it was caused by the dilithium crystals overheating.