fbpx
AnalysisOptus Outage Analysed

Optus Outage Analysed

Optus Outage Analysed By SEN’s Network & Communications Engineer, Chris Olsen.

-

- Advertisment -
  • HihkVision Colour Vu
  • HIKVISION
  • HID

Optus Outage Analysed
O

Optus Outage Analysed By SEN’s Network & Communications Engineer, Chris Olsen.

Optus Outage Analysed – It’s been reported by Matt Tett, the managing director of technology testing company, Enex TestLab, that there has only ever been a telecommunications outage of the magnitude of the recent Optus failure 3 or 4 times in the past 30 years.

And the question on the lips of all and sundry is: “What was the real cause of the problem?”.

When the CEO of Optus, Kelly Bayer Rosmarin, was recently interviewed on local radio, she was asked this very question. Her response: “The problem is too technical to explain”.

Bayer Rosmarin now faces the realistic possibility of a $A4 billion compensation bill, a review by the Australian Communications and Media Authority, and an official senate enquiry; such are the woes of most elevated.

Optus Outage Analysed By SEN’s Communications & Network Engineer

SEN was curious to try and pinpoint in greater detail what really caused upward of 10 million homes and over 400,000 Australian businesses to lose both internet and telecommunications connectivity for up to 16 hours last Wednesday. We investigated deeper, pushing past the general non-answer provided by Optus.

It turns out that the cause of the network shutdown was that its core routers received incorrect settings from one of the company’s overseas partners – the latest reports suggest this was Singtel – as part of a software firmware upgrade, causing a cascading failure; also known as flooding.

Those of you who remember IRC hacking and phreaking back in the 80s and 90s will have some idea how this works. It’s thought this incident was not born of a malevolent cyber security actor, though perhaps it’s too early to rule that out entirely.

It’s believed that the exact firmware fault can be tied to a BGP (border gateway protocol) prefix flood. Most of us in the computer or networked security industry have heard about BGP. In essence, it’s a protocol that routes data in a least-cost fashion to the closed next hop.

Optus Outage Analysed 2 LR
Optus Outage Analysed By SEN’s Communications & Network Engineer, Chris Olsen.

In this case, the firmware update or change broke BGP and caused it to route data through every path, instead of via the shortest path. As the firmware cascaded through the network, it opened the floodgates on each device, creating a virtual tsunami of data.

The routing table changes in the update propagated through multiple layers in the network and exceeded preset safety levels on the layer 2 routers. As the internal safety mechanisms on these routers were triggered, the only way for them to protect themselves was to disconnect from the Optus IP Core network.

To resolve the issue as quickly as possible, technicians had to physically travel to each affected device of which there are hundreds – some in third-part data centres – and manually revert the firmware to the previous version using a console cable and laptop. Thus, the 16-hour delay for network resurrection.

What can be done to mitigate such risks in the future? Avoiding single-point-of-failure network architecture should be considered when designing core networks to reduce the risk of total network outage. But sometimes, building a backup network is like asking the government to build a duplicate highway in case of an accident – it’s simply impractical.

That means security people need to factor communications redundancy and failover into their system designs.

Optus Outage Analysed By SEN’s Communications & Network Engineer

When relying on third parties to supply updates to core systems, it may be a better idea for national telcos to employ a small network engineering team to test third-party firmware before pushing it out across the whole network. The minor expense probably justifies the costs that will be incurred by an incident like this massive outage.

As an aside, it was reported on Thursday that Vodaphone had been buzzing with customers all day, as a 4-fold increase in activity was detected on its networks. Meanwhile, the Telstra Boost network saw a 5-fold increase in daily sales, while Kogan reported its sales of e-sims had increased by 400 per cent.

As former Prime Minister Malcolm Turnbull stated in relation to the incident, “Please note this as an example of how not to handle a crisis”.

You can see Optus’ response to the outage here or read more SEN news here.

“Optus Outage Analysed By SEN’s network engineer, Chris Olsen.”

Optus Outage Analysed 3 LR
Optus Outage Analysed – A classic firmware death stare.
Christopher Olsen
Christopher Olsenhttps://resolvit.com.au
Christopher shares with us valuable insights and industry insider tips to help improve readers' business operations. Bringing over 28 years of experience to the table in the IT, Legal and Business Operations & Systems sectors, Christopher looks forward to answering any of your questions, so please don't hesitate to post a comment in the thread below. Thanks for reading SEN News, and we look forward to bringing you more cutting-edge news about the security and technology industries.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

TRENDING

Price Hikes for Most Optus Postpaid Mobiles

Price Hikes for Most Optus Postpaid Mobiles Aligning with many other elements within the current economic climate, Optus has increased its post-paid mobile phone pricing...

Zigbee Technology

Zigbee Technology is today found throughout most homes worldwide and has become a standard in the wireless network infrastructure that runs building automation. It...

Latest news

Hikvision DS-K2604 Access Control Panel Quick Review

Hikvision has released economical and full-featured 2 and 4-door access control solutions, supported by a suite of devices, from keypads and biometrics readers to video intercoms, which integrate seamlessly with iVMS4200.

New Mercury MP Controller Line

New Mercury MP Controller Line Improves Flexibility And Cyber Security. New Mercury MP Controller Line — Mercury Security has announced...

IC3 Super West AI Data Centre Breaks Ground

IC3 Super West AI Data Centre Breaks Ground At MacQuarie Park. IC3 Super West AI Data Centre Breaks Ground -...
Click to Bookmark Post
Post Bookmarked
- Advertisement -

Preventative Maintenance Of Alarm Systems

Preventative Maintenance Of Alarm Systems To Reduce Expensive Call Outs. Preventative Maintenance Of Alarm Systems - I’d like to be...

Optic Security Integration Wins Award

Optic Security Integration Wins Award For Advanced Solution At Logistics Provider. Optic Security Integration Wins Award - Optic Security Group’s...

Dormakaba Axessor Apexx IP Safe Lock

Dormakaba Axessor Apexx IP Safe Lock Allows Enterprise Safe Lock Management. Dormakaba Axessor Apexx IP Safe Lock – One of...

Price Hikes for Most Optus Postpaid Mobiles

Price Hikes for Most Optus Postpaid Mobiles Aligning with many...

Zigbee Technology

Zigbee Technology is today found throughout most homes worldwide...

VMware Price Increases – Abandon Ship?

VMware Price Increases - Abandon Ship? VMWare's 100 per...

Open Source Software Explained Easily

Open Source Software Explained - This is generally those...
- Advertisement -

Dahua Technology Announces NZ Partnership

Dahua Technology Announces NZ Partnership – Dahua Technology New Zealand has announced Clear Digital as its new distribution partner in New Zealand.

Victoria Police Seeks Security Panel

Victoria Police Seeks Security Panel Victoria Police Seeks Security Panel To Deliver Security Services. Victoria Police Seeks Security Panel – Victoria...

Hikvision DS-K2604 Access Control Panel Quick Review

Hikvision has released economical and full-featured 2 and 4-door access control solutions, supported by a suite of devices, from keypads and biometrics readers to video intercoms, which integrate seamlessly with iVMS4200.

NAS Partners With Dahua Technology Australia

NAS Partners With Dahua Technology Australia. NAS partners with Dahua Technology Australia – NAS Australia, a member of the Security...

Is Soldering Best In Tough Environments

Is Soldering Best In Tough Environments Or Would You Recommend Alternative Methods? Is soldering best in tough environments - Would...

Proximity Access Readers: 125kHz or 13.56Mhz?

Proximity Access Readers: 125kHz or 13.56Mhz? Would you use 125kHz readers and cards or do you think 13.56Mhz is...
- Advertisement -

You might also likeRELATED
Recommended for you