Reset router could have saved Census

A single misconfigured router was the cause of August’s Census system outage, and the issue could have been resolved with the simple act of turning it off and on again.

The Senate committee inquiry into the handling of the August Census has heard that one of two routers used for the online Census system connecting to the Telstra network had been incorrectly coded so that when it was turned off in the wake of a DDoS attack it reset to a default state when switched back on.

After being powered back on, the router incorrectly reported information suggesting that Census data was being compromised by hackers, leading to the ABS’s decision to pull the site offline on 9 August.

IBM Senior Engineer Michael Shallcross acknowledged this error could have been discovered if the router was turned off and on again, the ABC reported. The company simulated the impact of a router failure but did not manually power it down.

Nobody at IBM has been fired or disciplined over the incident, according to the testimony. IBM is negotiating paying potential compensation to the ABS over the outage. The ABS has estimated that the outage will add $30 million to the cost of the Census.

Committee member liberal senator Jane Hume questioned whether IBM should have been able to anticipate the issue. “That’s the sort of level of technical competence I have with my computer,” she said.

IBM has also criticised Australia’s NextGen, one of the two ISPs tasked with controlling web traffic for the system, accusing the company of failing to prevent a DDoS attack from Singapore.

But in its own submission, NextGen said it had offered an alternative DDoS protection proposal to IBM for the project but had been turned down.

The Prime Minister’s special adviser on cybersecurity, Alistair MacGibbon, has meanwhile been critical of the system’s failure to handle the DDoS attacks. He noted that the attack was only around 3 Gbps, whereas DDoS attacks of well over 100 Gbps are regularly being mounted.

According to the Sydney Morning Herald, MacGibbon also identified the bureau’s backup plan in the event of a DDoS attack as fundamentally flawed.

The “Island Australia” plan to geoblock traffic to the site from overseas did not take into account the fact that the password reset utility used by IBM for the system was hosted offshore, and that geoblocking would have prevented the utility from working.

MacGibbon’s investigation also found that ABS staff misread a system report that gave them the impression that Census data was being compromised and led to the decision to shut down the site, and that there was a degree of vendor lock-in involved with ABS selecting IBM for the project.

ABS Chief David Kalisch meanwhile apologised for the inconvenience caused by the outage and pledged that the next Census in 2021 will go far more smoothly.

He defended the decision not to hold an open tender for the project, but said one would have been conducted had IBM not been able to satisfy the bureau that it could meet all the project’s requirements.

Image courtesy ABS.