DBS, POSB internet banking services restored after hours-long disruption
SINGAPORE: DBS and POSB internet banking and payment services have been restored after they went down for several hours on Thursday (May 2).
Services returned to normal at about 8pm, DBS said in an update on Facebook at about 8.15pm.
The bank said at 6.54pm – about an hour after the outage was first reported by users – that it was aware of the issues.
"We have identified the issue and have activated measures to recover the services," said the bank. "You can continue to use your DBS/POSB credit or debit cards to make payment."
It added that people could also use the ATMs nearest to them.
DBS Wealth clients could contact their relationship managers to place trades, while Vickers and mTrading customers should contact the Vickers hotline at 63272288 for help.
"Please be assured that your monies and deposits remain safe. We are sorry for the inconvenience caused," it added.
Downdetector showed that users started reporting disruptions with DBS internet banking services at about 5.45pm.
In a notice to users on its mobile app, DBS said: "Access to digital services is currently unavailable. We are resolving the issue and will update as soon as services are recovered."
In another error message, it said that it was experiencing heavy traffic to its services and urged people to login later.
Users commented on the bank's latest Facebook post about the latest outage.
In the first of these comments at about 5.58pm, a user said: "Your iBanking is down pls fix it!"
Customers reported being unable to access services on the DBS' mobile app and website platforms. Another user reported that he could "log in but can't make any transfer".
Many users also described their frustration at being unable to pay for food or drinks during dinner time.
The latest disruption comes two days after the Monetary Authority of Singapore (MAS) said that it would not seek to extend the six-month pause on the bank's non-essential activities.
MAS added that it will retain the multiplier of 1.8 times for DBS' risk-weighted assets for operational risk until the bank "has demonstrated the ability to maintain service availability and reliability, and handle any disruptions effectively".
The authority noted on Tuesday that DBS has since made "substantive progress" to address the shortcomings identified from the service disruptions last year, even as efforts to implement its remediation plan are ongoing.
MAS said it would closely monitor the bank's progress and the effectiveness of the measures.
"In the event of service disruptions, MAS expects DBS to promptly recover its services and communicate to its customers in a clear and timely manner," it said.
DBS said on Tuesday that being able to resume its activities "will not dilute its focus on strengthening technology resiliency and enhancing digital service availability".
DBS chief Piyush Gupta's variable pay cut by 30% over digital disruptions
SINGAPORE: Mr Piyush Gupta, the DBS chief executive officer, had his variable pay cut by 30 per cent in 2023, as a result of the digital disruptions experienced by the bank's customers.
This amounted to S$4.14 million (US$3.08 million), announced DBS on Wednesday (Feb 7) as part of its quarterly earnings statement. Collectively, the CEO and other members of the group management committee took a 21 per cent reduction in their variable pay.
"The Board determined that the variable compensation for the CEO and other members of the Group Management Committee should be cut to hold them accountable for the series of digital disruptions during the year," DBS said.
Singapore's largest lender suffered a series of disruptions to its digital banking services last year, culminating in the Monetary Authority of Singapore (MAS) barring DBS from any acquisitions of new business ventures for six months.
DBS was also made to pause non-essential IT changes for six months and was not allowed to reduce the size of its branch and ATM networks in Singapore.
Q4 EARNINGS
The bank maintained guidance for net interest income for 2024 at around last year's levels after posting a 2 per cent rise in fourth-quarter net profit, beating expectations.
"While interest rates are expected to soften and geopolitical tensions persist, our franchise strengths will put us in good stead to sustain our performance in the coming year," Mr Gupta said in a statement.
Besides maintaining net interest income at around 2023 levels, he expected return on equity (ROE) to be 15 per cent to 17 per cent for this year and fee income growth at double-digit, according to slides accompanying his results.
Full-year net interest margin (NIM), a key profitability gauge, is expected to be slightly below fourth quarter NIM of 2.13 per cent.
Singapore's banks, the largest in Southeast Asia, are set to post higher profits for the fourth quarter because of higher interest rates, though growth momentum is poised to slow as central banks pivot toward rate cuts and volatile markets weigh on the wealth business.
DBS, the first Singapore lender to report this earnings season, said October-December net profit grew to S$2.39 billion (US$1.78 billion) from S$2.34 billion a year earlier on the back of a 9 per cent increase in total income.
This beat the mean estimate of S$2.37 billion from four analysts, according to LSEG data.
DBS, which is also Southeast Asia's biggest bank, proposed a final dividend of 54 cents per share and 1-for-10 bonus issue.
The NIM of 2.13 per cent during the quarter was up from 2.05 per cent a year earlier.
Full-year annual profit jumped 26 per cent to S$10.3 billion from S$8.19 billion in 2022. Return on equity climbed to a record high of 18 per cent from 15 per cent a year ago.
NUS Professor calls for stronger oversight on DBS CEO’s pay
SINGAPORE: In a Feb 14 (Wednesday) commentary for CNA, NUS Business School Professor of Accounting Mak Yuen Teen weighed in on theS$4.14 million variable pay cut taken by DBS CEO Piyush Gupta due to service outages last year that had been announced on Feb 7.
Prof Mak, who specialises in corporate governance, wrote that banks could be more transparent when it comes to senior management salaries.
Policies surrounding the pay of senior management and rank-and-file employees should be “subject to appropriate oversight by a truly independent committee,” he wrote, adding that “If the remuneration committee and the board approve a policy that drives the wrong behaviour, then they should bear some responsibility.”
After disruptions to its digital banking services inconvenienced many of its customers last year, DBS said its senior management would be held accountable for these shortcomings and that this accountability would be reflected in their remuneration.
The bank also issued a public apology and said it would use an additional S$80 million to fortify its system resilience.
On Feb 7, The Straits Times reported that Mr Gupta took home S$15.4 million in 2022. Prof Mak pointed out that this made him the highest-paid bank CEO in Asia Pacific for the year.
Assuming that the CEO’s base salary and benefits are unchanged, he likely took home S$11.26 million in 2023, though the exact figure will be known when the bank releases its annual report next month.
Prof Mak traced Mr Gupta’s salaries over the past few years, saying that in 2021, after a two-day service disruption in November, he took home S$13.6 million.
This meant that his remuneration for that year was almost 50 per cent higher than in 2020, although, like at many other banks, top management took a pay cut due to the Covid-19 pandemic that year.
Mr Gupta’s remuneration, including variable pay, has been increasing since 2019, and the 2021 disruptions were not mentioned in the assessment of his performance in the bank’s annual report, Prof Mak added. The CEO was even lauded for leading the bank to its “best year ever in 2021.”
“Mr Gupta’s significant pay cut this year is likely the result of the board exercising its discretion to adjust his remuneration, rather than him not meeting specific key performance indicators (KPIs),” wrote the NUS professor.
He went on to compare how the compensation packages for the CEOs in Singapore’s biggest banks are computed with how this is done in Australia, where “Australian banks’ CEOs were also paid consistently less than their Singaporean counterparts.”
Prof Mak also wrote that when things have gone “badly wrong” in Australian banks, there is a greater degree of accountability, with some CEOs even losing their jobs. The same has happened with banks in Canada and Japan.
Even though there has not been “widespread misconduct” as in the case of Australian banks, he added that there are important takeaways from the findings of the Royal Commission that could be relevant to the local situation.
“First, balanced scorecards that are used by banks may not be achieving their objectives.
Second, there is a need to review remuneration policies at all levels and in different functions.
Third, and most importantly, culture, governance and remuneration are key causes of misconduct – and they are inextricably linked,” wrote Prof Mak, adding:
“Given the significant reliance of our local banks on variable pay to reward their senior management, they need to ensure that their remuneration policies drive the right behaviour.”
Prof Mak traced Mr Gupta’s salaries over the past few years, saying that in 2021, after a two-day service disruption in November, he took home S$13.6 million.
This meant that his remuneration for that year was almost 50 per cent higher than in 2020, although, like at many other banks, top management took a pay cut due to the Covid-19 pandemic that year.
Mr Gupta’s remuneration, including variable pay, has been increasing since 2019, and the 2021 disruptions were not mentioned in the assessment of his performance in the bank’s annual report, Prof Mak added. The CEO was even lauded for leading the bank to its “best year ever in 2021.”
Indian Bank Accidentally Credits S$132M Into Customer Accounts After Technical Issue
A technical issue recently caused UCO, a bank in India, to credit about 8.2 billion rupees (S$132 million) into customers’ accounts by accident.
The bank immediately froze the affected accounts upon discovering the error.
Currently, they have recovered about 6.49 billion rupees (S$104 million), which is about 79% of the erroneous transfers.
India’s government has since directed state-owned banks to improve digital security following the incident.
Technical issue causes ‘erroneous credits’ into some UCO accounts in India
According to The Hindu Business Line, UCO revealed that an internal technical issue resulted in some customers receiving “erroneous credits” via its Immediate Payment Service (IMPS).
The IMPS allows immediate electronic fund transfers via mobile or internet banking, similar to Singapore’s PayNow system.
Reportedly, the error occurred between 10 and 13 Nov.
Certain transactions from account holders of other banks had apparently gone to UCO Bank accounts as a result of the glitch.
In spite of that, the UCO account holders did not actually receive any money.
“We wish to clarify that there was an issue with the IMPS platform,” UCO stated.
DBS, Citi outages prevented 2.5 million payment and ATM transactions from being completed
SINGAPORE — Around 2.5 million payment and ATM transactions could not be completed during last month’s DBS and Citibank service disruptions, Minister of State for Trade and Industry Alvin Tan said in parliament on Monday (Nov 6).
Up to 810,000 attempts to access the digital banking platforms of the two banks also failed between 2.54pm on Oct 14 and 4.47am the next day, said Mr Tan, describing the impact of the outage as “wide”.
DBS and Citibank’s online banking and payment services were disrupted, while DBS automated teller machines (ATMs) at several locations were down. The outage was caused by a “technical issue” with the cooling system at an Equinix data centre.
The temperature at the data centre rose above the optimal operating range, causing the information technology (IT) systems of both banks to shut down, Mr Tan said.
Mr Tan added DBS and Citibank immediately activated their IT disaster recovery and business continuity plans.
“However, both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup data centres,” he said.
DBS was affected by a network misconfiguration and Citibank was affected by connectivity issues.
Services at DBS and Citibank were progressively restored from 8.21pm and 7.05pm respectively on Oct 14, but only fully recovered in the early hours of Oct 15.
DBS and Citibank fell short of the Monetary Authority of Singapore (MAS) requirements to ensure that critical IT systems are resilient against prolonged disruptions, Mr Tan said.
MAS requires banks to establish IT disaster recovery plans and test them regularly. Banks must conduct disaster recovery exercises with their backup data centres to check that critical systems and services can be restored within four hours of an outage.
“While both banks conducted annual exercises to test the recovery of their IT systems at their backup data centres, the specific issues that led to the delays in system recovery on Oct 14 did not surface during those tests,” Mr Tan said.
PENALTIES IMPOSED ON DBS
Last week, MAS announced that DBS would not be allowed to make non-essential IT changes or acquire new business ventures for six months.
Analysts told CNA that they were unaware of upcoming acquisition plans for DBS. Mr Thilan Wickramasinghe, head of Singapore research at Maybank, said DBS was focused on integrating other businesses into the bank and likely has “limited appetite” for more mergers and acquisitions.
MAS also said DBS must maintain the size of its branch and ATM networks for now, and continue to apply a multiplier of 1.8 times to its risk-weighted assets for operational risk.
It was first instructed to do so in May. At that time, the multiplier of 1.8 times translated to around S$1.6 billion in additional regulatory capital, or money that must be set aside as a buffer.
A higher capital requirement can limit the bank’s ability to invest.
“Holding additional regulatory capital comes with costs for the bank. It increases cost of capital, a key metric that drives business decisions such as dividends and investments," said Mr Tan.
“It is a drag on the return on capital which could in turn impact credit ratings and stock price of the bank."
TECHNOLOGY IS FALLIBLE
While financial institutions have to take prompt action in the event of an outage to reduce inconvenience to customers, users should also be prepared for disruptions, Mr Tan said.
“No IT system is infallible. Disruptions can occur for a variety of reasons and can happen without warning,” he said.
During the Oct 14 outage, customers who were able to switch to alternative payment methods or providers, or had cash on hand were less affected, he said.
“While we want to be digital first (in) our approach to digitalisation, we cannot be digitally only,” said Mr Tan.
Additionally, consumers and businesses should be aware that some people may not be able to transition to digital payments smoothly.
"Some people may not be adept at technology, and banks and financial institutions, and in fact companies, service providers must acknowledge that, and must be able to provide other forms of payment approaches so that others who are not so adept can still participate in the economy," he said.
Piyush Gupta: Asia’s most disrupted digital banker
It’s a jarring moment. Piyush Gupta, CEO of DBS Group is often lauded as Asia’s top financier and has been tipped as a candidate to lead global rivals. The $62 billion Singaporean bank is also held up as a role model by Chinese insurer Ping An in its shareholder agitation at HSBC. Yet following repeated serious failures in its digital banking services, the Monetary Authority of Singapore has banned DBS from non-essential activities including M&A. It puts dividends at risk and will attract the eye of watchdogs overseas where the lender is busy expanding.
It undermines Gupta’s championing over the past decade of DBS as a technology company and adds to a monetary censure slapped on the bank. In May, it was ordered to retain a multiplier of 1.8 times to its risk weighted assets for operational risk, an increase from a multiplier of 1.5 times applied in February 2022. The higher figure translates to approximately S$1.6 billion, equivalent to nearly $1.2 billion in total additional regulatory capital, the regulator said at the time. Though the latest punishment is not financial, there are financial ramifications.
First, fixing the technology problems will require spending. What’s more, MAS has warned it could increase the capital requirements depending on DBS' so far uninspiring progress. Those factors will dampen investors’ expectations of a special dividend after its yearend earnings; the stock fell nearly 1% on Thursday. If dividends aren’t raised in line with Citi’s forecasts, analysts there reckon DBS is trading at 5.8% dividend yield for 2023 compared to compatriot OCBC’s visible path to 6.4%.
The bigger issue for the bank is reputational. The digital transformation smoothed DBS’ foray into large markets such as India, where the regulator in 2020 handpicked it to rescue a crisis-hit Lakshmi Vilas Bank, a local lender with over 500 branches. While that integration and the Singaporean bank’s separate purchase of Citi’s Taiwan consumer business last year are complete, regulators there are likely to pay closer attention to DBS. And whenever it is allowed to resume dealmaking, the bank may also find a cooler reception.
DBS Chairman Peter Seah warned that senior management would be held accountable for the tech lapses including on pay. On the back of record profits, Gupta earned $11.4 million last year, making him one of Asia’s highest remunerated bank CEOs. He told Breakingviews in September that he is more interested in becoming a naturalist than taking up another banking role whenever he leaves the lender he’s led since 2009. The shareholder returns Gupta has delivered remain enviable but his tech stripes look less impressive.
Service outages: MAS bars DBS from acquiring new business ventures, reducing branch and ATM network sizes for 6 months
• DBS has been barred from new business venture acquisitions for six months after repeated disruptions to its services in 2023 so far
• The Monetary Authority of Singapore said that the bank is not allowed to reduce the size of its branch and automated teller machines networks in Singapore
• DBS’ non-essential information technology changes have also been paused during this six-month period
• Its chairman Peter Seah said that the senior management will be held accountable for the disruptions and it will be reflected in their compensation
• MAS said that it will review the bank’s remediation efforts at the end of the stipulated period
SINGAPORE — DBS bank has been barred from acquiring new business ventures by the Monetary Authority of Singapore (MAS) and instructed not to reduce the size of its branch and automated teller machines (ATM) networks in Singapore.
In a media statement on Wednesday (Nov 1), the central bank said that during this stipulated six-month period, DBS is also required to pause its non-essential information technology changes.
The actions imposed by the authority were taken after repeated and prolonged disruptions of the bank’s services so far this year, and they are meant to ensure that the bank keeps “sharp focus on restoring the resilience of its digital banking services', MAS added.
They were unable to access many services provided by the bank, including the ATMs and fund transfers via online payment services PayNow and PayLah!, as well as payments at physical payment terminals such as those in stores and eateries.
DBS said then that the disruption was caused by an issue at a common data centre, Equinix, which was also used by other organisations.
Earlier in the year, DBS' customers were subject to disruptions in Marchand May.
After the March incident, MAS said that it directed DBS to engage an independent third-party to conduct a comprehensive review of the effectiveness and adequacy of the people, processes and technology supporting its digital banking services.
Several shortcomings were later identified, namely on the bank’s system resilience, incident management, change management and technology risk governance and oversight.
MAS said on Wednesday: “Following the independent review, DBS has set out a technology resiliency roadmap to address the shortcomings, improve system resilience, and better position the bank to meet future digital banking needs.
“The roadmap is being implemented in phases, with the changes affecting its system architecture design taking more time to complete.”
MAS, which reviewed DBS’ remediation plan under the roadmap, said that it was “satisfied with its scope” and the planned measures to improve system resilience.
It added: “In line with MAS’ expectations, DBS will hold senior management accountable for the lapses and the board will enhance its governance approach to oversee implementation of the roadmap.
“MAS has directed DBS to suspend all changes to the bank’s IT systems except for those related to security, regulatory compliance and risk management for a six-month period.
“This is to ensure that the bank dedicates the needed resources and attention to strengthen its technology risk management systems and controls. MAS will not approve any new business acquisitions by the bank during this period.”
ATM NETWORK AND BRANCH SIZES
The bank was also directed not to reduce the size of its branch and ATM networks to ensure that there are enough alternative channels for its customers in the event of further disruptions while the bank works to enhance the operational resilience of its digital channels.
This direction will be in force until MAS is satisfied with the progress of DBS’ remediation plan.
MAS said that it will review the bank’s remediation efforts after six months and it may then extend the duration of the measures, vary the additional capital requirement now imposed, or take further actions at that point.
“In the meantime, MAS will retain the multiplier of 1.8 times to DBS’ risk weighted assets for operational risk, which was imposed after the March and May 2023 incidents. “
DBS will take up to 24 months to put in place the planned structural changes to improve the resilience of its digital banking services, MAS said.
“In the meantime, it is possible that disruptions may still occur. In such situations, MAS expects DBS to promptly recover its services and communicate to its customers in a clear and timely manner,” it added.
MAS’ deputy managing director of financial supervision Ho Hern Shin said: “DBS must put in place immediate measures to ensure service reliability while it continues to invest in the longer-term efforts to bolster its operational resilience.
“We have imposed this six-month pause on the bank to give it the space to take the actions needed to maintain customer trust.
SENIOR MANAGEMENT WILL BE HELD ACCOUNTABLE
DBS'chairman Peter Seah issued an apology on Wednesday for the digital banking disruptions that happened this year.
He said: “The board apologises for the digital banking disruptions. When customers bank with us, they expect to be able to access our banking services conveniently, and at any time of the day.
“With the incidents of the past year, we have failed to live up to these expectations, and have also fallen short of our own standards.”
Mr Seah added that the senior management will be held accountable as an acknowledgment that the bank could have done better, adding that this will be reflected in their compensation.
DBS' chief executive officer Piyush Gupta said on Wednesday that the bank will be setting aside a special budget of S$80 million to enhance system resiliency.
"Over the years, DBS has focused on digital transformation so as to make banking simple, seamless and effortless," he added.
"However, we acknowledge that we must now do better to deliver on this, and are taking a multitude of actions across technology governance, people, leadership, systems and processes."
CIO Jimmy Ng gets "demoted", Piyush Gupta however walked away scot-free tsk tsk
DBS is also streamlining its technology teams. Chief Information Officer Jimmy Ng, who also heads the group’s technology & operations, will only be in charge of operations. The bank is searching for a new CIO. Singapore country head Han Kwee Juan will take an additional role of acting CIO. Sim S. Lim, a senior adviser, will temporarily return to day-to-day functions of the home-turf market.
'This is not supposed to happen': Experts on DBS, Citi outage caused by data centre failure
Most banks have two or more data centres and redundancies at multiple levels if a primary data centre goes offline, industry experts say.
SINGAPORE: Banks are heavily reliant on data centres for their operations, making it crucial to have a backup data centre should one fail, banking and technology experts told CNA in the wake of a DBS and Citibank outage earlier this month.
These centres form the nerve centre of the banking ecosystem, and they store, manage and process massive amounts of data, the experts said.
In the case of DBS and Citibank, the data centre they used experienced technical issues with its cooling system during a planned upgrade, raising temperatures and affecting equipment. The incident disrupted both banks' payment and banking services on Oct 14.
Last Friday, the banks’ data centre provider Equinix said its initial investigation found that the incident was caused by a vendor contractor. The contractor had incorrectly sent a signal to close the valves from the chilled water buffer tanks, which affected the flow of chilled water to the cooling system.
Equinix said it is reviewing its processes and putting in place additional audits to prevent a similar incident during future upgrades.
The DBS and Citibank outage started on Oct 14 at 3pm and only fully resumed the following morning. Customers were unable to access both banks' apps and online banking or payment services such as PayLah! and PayNow. ATM services were also down at several locations.
Last Thursday, the Monetary Authority of Singapore (MAS) said it had ordered both banks to conduct a thorough investigation, noting that the downtime exceeded its limit of four hours within a 12-month period.
While MAS has no oversight over data centres, it expects banks to establish contractual agreements with data centre providers that incorporate its requirements on system availability, it said.
DATA CENTRES THE "NERVE CENTRE"
Data centres and their functions form the backbone of modern-day businesses, said Associate Professor Lee Poh Seng.
"At their core, they house a variety of components, including servers, storage systems, networking hardware like switches and routers, and the cabling infrastructure necessary for data and network communications,” said Assoc Prof Lee, who is from the National University of Singapore’s (NUS) Department of Mechanical Engineering.
Singapore Management University's (SMU) Dr Patrick Thng, who has close to three decades of experience in banking, including as managing director at various banks, likened data centres to a car's engine.
"From very fundamental office support applications … to banking critical operations, like core banking, credit card applications, ATM, foreign exchange trading, branch platform, digital banking services, (all rely on data centres)," said the director of a financial technology and analytics programme at SMU.
"Data centres have massive storage to store all your transactions and customer data. Banks rely on this data centre like a nerve centre."
COOLING SYSTEM CRITICAL TO DATA CENTRE
NUS’ Assoc Prof Lee pointed out that cooling systems are critical to data centres. Equinix uses a chilled water system, which works by circulating chilled water to absorb heat from the data centre environment and dissipate it outside the facility.
“They keep the hardware at optimal operating temperatures, and a failure could result in overheating and subsequent IT hardware failures,” he said.
According to the Singapore Computer Society, most data centres use centralised chiller plants that have built-in redundancy to enable continuous cooling in the event of equipment failure. Outages still occur, but are “very rare”, it said.
The recommended temperature range for data centres is 18 to 27 degrees Celsius, said Assoc Prof Lee, referring to guidelines by the American Society of Heating, Refrigerating and Air-Conditioning Engineers.
In Singapore’s tropical climate, with high year-round temperatures and humidity, robust cooling systems to prevent overheating are necessary.
When a cooling system is down and temperatures are not brought back to a safe range, overheating can ensue. This can cause intermittent hardware errors, data corruption, and in severe cases, permanent hardware damage, added Assoc Prof Lee, who holds two US patents in thermal systems.
Asked why it might have taken such a long time for the data centre to resume operations, Assoc Prof Lee said: “If not previously encountered, the incident's nature could present a learning curve, extending the recovery time. The adequacy and effectiveness of disaster recovery procedures in place could also affect the recovery time.”
BACKUPS, RECOVERY PLANS NEEDED
The importance of the data centre means that banks always have a backup data centre. Some banks also have two data centres that share the workload concurrently. In such cases, if one fails, the other can pick up the slack.
Dr Dennis Khoo, a managing partner at digital consultancy allDigitalFuture, said: "Generally for mission-critical applications like banks, there are multiple levels of redundancy.
"In most advanced banks, using the latest technologies, the database will be instantly replicated, that means they'll have a primary site and alternate site and the data is replicated instantly on both sites," said Dr Khoo.
The Singapore Computer Society said most data centres are designed and built with a certain level of redundancy and the ability to conduct maintenance in real time. They are also specifically built to meet the exact redundancy requirements of the business, it said.
A common uptime guarantee of a data centre would usually be at 99.982 per cent to mitigate possible disruption to its customers, the society added.
However, there is still 0.018 per cent of possible downtime. “Thus, the client must establish an efficient Business Continuity Management System and IT Disaster Recovery Plan to allow their critical IT systems and data information to immediately failover to the secondary data centre in the shortest possible time should such an incident occur.”
WHAT CAN BANKS DO WHEN DATA CENTRES ARE DOWN
That said, should all data centres fail, there is little – if any – service a bank can provide.
Dr Thng said banks activate what they call "offline mode", which means they render some services at branch offices using what is available. These transactions are then updated with the mainframe when the data centre is back up.
These services may include cash withdrawals – as banks have spare cash on hand – cash deposits, payment instructions and credit card transactions. In the case of DBS, it reopened branches to help customers with some services.
According to Dr Thng, IT incidents, such as a delay in processing, occur daily. Some of these go unnoticed by customers.
Still, service outages will negatively impact the bank’s reputation and may have some financial repercussions, for example, if a client incurs late fees from being unable to pay his bill on time due to an outage.
On reputational impact, Dr Khoo said banks would have “broken” their service commitments to provide around-the-clock service to customers.
“So definitely, in that sense, there will be some reputational damage in terms of your ability to serve customers properly. And with proper design, this is not supposed to happen.”
Bloomberg: India producing graduates with worthless degrees
Bloomberg reported that India’s US$117 billion education industry is producing graduates with worthless degrees and an unemployed generation in India.
Desperate for jobs, some of these Indian graduates are paying for two or three degrees. Many are attending “colleges” popping up inside small apartment buildings or inside shops in marketplaces.
Bloomberg found out that many of these small private colleges, in fact, don’t have regular classes, employ teachers with little training, use outdated curriculums, and offer no practical experience or job placements.
One of the students told Bloomberg that it was easy to secure admission and get a degree without attending class.
Tanmay Mandal, 25, told Bloomberg that he paid US$4,000 for a bachelor’s degree in civil engineering. He ended up learning almost nothing from teachers who appeared to have insufficient training themselves. He couldn’t answer technical questions at job interviews, and has been unemployed for the last three years.
“Many of my friends are also sitting idle without a job,” said Mandal.
Pankaj Tiwari, 28, said he paid 100,000 rupees (US$1,200) for a master’s degree in digital communication because he wanted a job.
Though his college had promised campus placements, no company turned up and he’s still unemployed four years later. “I feel like I wasted my time,” he said. “I just secured certificates on paper, but those are of no use.”
According to a study by talent assessment firm Wheelbox, half of all graduates in India are unemployable due to problems in its education system.
Anil Swarup, a former secretary for school education estimated that of 16,000 colleges handing out bachelor’s qualifications for teachers, a large number existed only in name and one study by the human resource firm SHL found that only 3.8% of engineers have the skills needed to be employed in software-related jobs.
Fake patients and fake degrees
Even some of the more reputable universities and colleges are engaging in dubious practices in order to maintain their accreditation.
In 2019, the Supreme Court barred the Bhopal-based RKDF Medical College Hospital and Research Centre from admitting new students for two years for allegedly using fake patients to meet medical college requirements.
The college initially argued in court that the patients were genuine, but later apologised after an investigative panel found that the purported patients weren’t really sick.
“We have noticed a disturbing trend of some medical colleges in projecting fake faculty and patients for obtaining permission for admission of students,” the court said in its judgement.
The medical school is part of RKDF Group, which has a wide network of colleges in areas from engineering to medicine and management.
In May last year, police arrested the vice chancellor of RKDF Group’s Sarvepalli Radhakrishnan University as well as his predecessor for alleged involvement in giving out fake degrees.
Still, students could be seen flooding into several of RKDF’s institutions undeterred.
String of institutions in various Indian states are also drawing scrutiny. In some parts of India, students have gone on hunger strikes protesting the lack of teachers and facilities at their institutes.
In 2017, one institution in the eastern state of Odisha gave fake job offers during campus placements leading to protests by students.
And in January this year, charges were finally filed against Manav Bharti University and its promoters for allegedly selling fake degrees.
Their alleged fraud was actually discovered in February 2021 when during a routine investigation of a fake degree case, the Indian police actually uncovered a much bigger case of Manav Bharti University having sold 36,000 fake degrees across 17 states in over 11 years.
At the time, TOC did a cursory check on LinkedIn and found that there were already a number of graduates from Manav Bharti University working in Singapore.
Later in July 2021, Singapore’s Manpower Ministry (MOM) released a statement saying that it had conducted an investigation into 23 foreign individuals with qualifications from Manav Bharti University.
Two Indian nationalswere jailed, and 19 others were permanently barred from future employment in Singapore for using fake degrees to apply for work passes.
In one case, an Indian national submitted his fake degree to MOM in his S Pass application in 2015 to work as an assistant warehouse manager in Singapore. MOM took some six years before discovering his degree was in fact a fake.
DBS digibank service disruptions affecting some iOS users on Jun. 1 morning already resolved: DBS
Intermittent service disruptions faced by DBS's digibank app on Saturday morning have since been resolved, said a bank spokesperson on Jun. 1.
Over 500 reports of network issues were made on service disruption tracking site Downdetector from 7am.
A DBS spokesperson stated that between 7:52am and 11:02am, some users were unable to access the digibank Mobile app.
Only a "small number of iOS users" were affected, the spokesperson added.
Other digital services, such as DBS Paylah, digibank Online (Internet Banking), and DBS/POSB credit cards remained usable at the time.
"We have since resolved the issue and all iOS users could access digibank Mobile normally from 11:02am," the spokesperson said.
DBS, POSB internet banking services restored after hours-long disruption
SINGAPORE: DBS and POSB internet banking and payment services have been restored after they went down for several hours on Thursday (May 2).
Services returned to normal at about 8pm, DBS said in an update on Facebook at about 8.15pm.
The bank said at 6.54pm – about an hour after the outage was first reported by users – that it was aware of the issues.
"We have identified the issue and have activated measures to recover the services," said the bank. "You can continue to use your DBS/POSB credit or debit cards to make payment."
It added that people could also use the ATMs nearest to them.
DBS Wealth clients could contact their relationship managers to place trades, while Vickers and mTrading customers should contact the Vickers hotline at 63272288 for help.
"Please be assured that your monies and deposits remain safe. We are sorry for the inconvenience caused," it added.
Downdetector showed that users started reporting disruptions with DBS internet banking services at about 5.45pm.
In a notice to users on its mobile app, DBS said: "Access to digital services is currently unavailable. We are resolving the issue and will update as soon as services are recovered."
In another error message, it said that it was experiencing heavy traffic to its services and urged people to login later.
Users commented on the bank's latest Facebook post about the latest outage.
In the first of these comments at about 5.58pm, a user said: "Your iBanking is down pls fix it!"
Customers reported being unable to access services on the DBS' mobile app and website platforms. Another user reported that he could "log in but can't make any transfer".
Many users also described their frustration at being unable to pay for food or drinks during dinner time.
The latest disruption comes two days after the Monetary Authority of Singapore (MAS) said that it would not seek to extend the six-month pause on the bank's non-essential activities.
CNA has contacted DBS for more information.
On Tuesday, MAS said that it would not seek to extend the six-month pause on non-essential activities that it imposed on DBS after the bank's multiple service disruptions in 2023.
MAS added that it will retain the multiplier of 1.8 times for DBS' risk-weighted assets for operational risk until the bank "has demonstrated the ability to maintain service availability and reliability, and handle any disruptions effectively".
The penalty was imposed in May 2023, up from the 1.5 times that was implemented in February 2022.
The authority noted on Tuesday that DBS has since made "substantive progress" to address the shortcomings identified from the service disruptions last year, even as efforts to implement its remediation plan are ongoing.
MAS said it would closely monitor the bank's progress and the effectiveness of the measures.
"In the event of service disruptions, MAS expects DBS to promptly recover its services and communicate to its customers in a clear and timely manner," it said.
DBS said on Tuesday that being able to resume its activities "will not dilute its focus on strengthening technology resiliency and enhancing digital service availability".
https://www.channelnewsasia.com/singapore/dbs-posb-internet-banking-services-down-users-payment-paylah-paynow-4308641
DBS = Diarrhoea Bank of Sinkapoor
DBS chief Piyush Gupta's variable pay cut by 30% over digital disruptions
SINGAPORE: Mr Piyush Gupta, the DBS chief executive officer, had his variable pay cut by 30 per cent in 2023, as a result of the digital disruptions experienced by the bank's customers.
This amounted to S$4.14 million (US$3.08 million), announced DBS on Wednesday (Feb 7) as part of its quarterly earnings statement. Collectively, the CEO and other members of the group management committee took a 21 per cent reduction in their variable pay.
"The Board determined that the variable compensation for the CEO and other members of the Group Management Committee should be cut to hold them accountable for the series of digital disruptions during the year," DBS said.
Singapore's largest lender suffered a series of disruptions to its digital banking services last year, culminating in the Monetary Authority of Singapore (MAS) barring DBS from any acquisitions of new business ventures for six months.
DBS was also made to pause non-essential IT changes for six months and was not allowed to reduce the size of its branch and ATM networks in Singapore.
Q4 EARNINGS
The bank maintained guidance for net interest income for 2024 at around last year's levels after posting a 2 per cent rise in fourth-quarter net profit, beating expectations.
"While interest rates are expected to soften and geopolitical tensions persist, our franchise strengths will put us in good stead to sustain our performance in the coming year," Mr Gupta said in a statement.
Besides maintaining net interest income at around 2023 levels, he expected return on equity (ROE) to be 15 per cent to 17 per cent for this year and fee income growth at double-digit, according to slides accompanying his results.
Full-year net interest margin (NIM), a key profitability gauge, is expected to be slightly below fourth quarter NIM of 2.13 per cent.
Singapore's banks, the largest in Southeast Asia, are set to post higher profits for the fourth quarter because of higher interest rates, though growth momentum is poised to slow as central banks pivot toward rate cuts and volatile markets weigh on the wealth business.
DBS, the first Singapore lender to report this earnings season, said October-December net profit grew to S$2.39 billion (US$1.78 billion) from S$2.34 billion a year earlier on the back of a 9 per cent increase in total income.
This beat the mean estimate of S$2.37 billion from four analysts, according to LSEG data.
DBS, which is also Southeast Asia's biggest bank, proposed a final dividend of 54 cents per share and 1-for-10 bonus issue.
The NIM of 2.13 per cent during the quarter was up from 2.05 per cent a year earlier.
Full-year annual profit jumped 26 per cent to S$10.3 billion from S$8.19 billion in 2022. Return on equity climbed to a record high of 18 per cent from 15 per cent a year ago.
https://www.channelnewsasia.com/business/dbs-piyush-gupta-pay-cut-digital-disruptions-4105246
DBS fouls up once again
CECAs cannot and must not be entrusted with the oversight of key IT infrastructure, period.
Happening soon in SG?
Indian Bank Accidentally Credits S$132M Into Customer Accounts After Technical Issue
A technical issue recently caused UCO, a bank in India, to credit about 8.2 billion rupees (S$132 million) into customers’ accounts by accident.
The bank immediately froze the affected accounts upon discovering the error.
Currently, they have recovered about 6.49 billion rupees (S$104 million), which is about 79% of the erroneous transfers.
India’s government has since directed state-owned banks to improve digital security following the incident.
Technical issue causes ‘erroneous credits’ into some UCO accounts in India
According to The Hindu Business Line, UCO revealed that an internal technical issue resulted in some customers receiving “erroneous credits” via its Immediate Payment Service (IMPS).
The IMPS allows immediate electronic fund transfers via mobile or internet banking, similar to Singapore’s PayNow system.
Reportedly, the error occurred between 10 and 13 Nov.
Certain transactions from account holders of other banks had apparently gone to UCO Bank accounts as a result of the glitch.
In spite of that, the UCO account holders did not actually receive any money.
“We wish to clarify that there was an issue with the IMPS platform,” UCO stated.
DBS, Citi outages prevented 2.5 million payment and ATM transactions from being completed
SINGAPORE — Around 2.5 million payment and ATM transactions could not be completed during last month’s DBS and Citibank service disruptions, Minister of State for Trade and Industry Alvin Tan said in parliament on Monday (Nov 6).
Up to 810,000 attempts to access the digital banking platforms of the two banks also failed between 2.54pm on Oct 14 and 4.47am the next day, said Mr Tan, describing the impact of the outage as “wide”.
DBS and Citibank’s online banking and payment services were disrupted, while DBS automated teller machines (ATMs) at several locations were down. The outage was caused by a “technical issue” with the cooling system at an Equinix data centre.
The temperature at the data centre rose above the optimal operating range, causing the information technology (IT) systems of both banks to shut down, Mr Tan said.
Mr Tan added DBS and Citibank immediately activated their IT disaster recovery and business continuity plans.
“However, both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup data centres,” he said.
DBS was affected by a network misconfiguration and Citibank was affected by connectivity issues.
Services at DBS and Citibank were progressively restored from 8.21pm and 7.05pm respectively on Oct 14, but only fully recovered in the early hours of Oct 15.
DBS and Citibank fell short of the Monetary Authority of Singapore (MAS) requirements to ensure that critical IT systems are resilient against prolonged disruptions, Mr Tan said.
MAS requires banks to establish IT disaster recovery plans and test them regularly. Banks must conduct disaster recovery exercises with their backup data centres to check that critical systems and services can be restored within four hours of an outage.
“While both banks conducted annual exercises to test the recovery of their IT systems at their backup data centres, the specific issues that led to the delays in system recovery on Oct 14 did not surface during those tests,” Mr Tan said.
PENALTIES IMPOSED ON DBS
Last week, MAS announced that DBS would not be allowed to make non-essential IT changes or acquire new business ventures for six months.
Analysts told CNA that they were unaware of upcoming acquisition plans for DBS. Mr Thilan Wickramasinghe, head of Singapore research at Maybank, said DBS was focused on integrating other businesses into the bank and likely has “limited appetite” for more mergers and acquisitions.
MAS also said DBS must maintain the size of its branch and ATM networks for now, and continue to apply a multiplier of 1.8 times to its risk-weighted assets for operational risk.
It was first instructed to do so in May. At that time, the multiplier of 1.8 times translated to around S$1.6 billion in additional regulatory capital, or money that must be set aside as a buffer.
A higher capital requirement can limit the bank’s ability to invest.
“Holding additional regulatory capital comes with costs for the bank. It increases cost of capital, a key metric that drives business decisions such as dividends and investments," said Mr Tan.
“It is a drag on the return on capital which could in turn impact credit ratings and stock price of the bank."
TECHNOLOGY IS FALLIBLE
While financial institutions have to take prompt action in the event of an outage to reduce inconvenience to customers, users should also be prepared for disruptions, Mr Tan said.
“No IT system is infallible. Disruptions can occur for a variety of reasons and can happen without warning,” he said.
During the Oct 14 outage, customers who were able to switch to alternative payment methods or providers, or had cash on hand were less affected, he said.
“While we want to be digital first (in) our approach to digitalisation, we cannot be digitally only,” said Mr Tan.
Additionally, consumers and businesses should be aware that some people may not be able to transition to digital payments smoothly.
"Some people may not be adept at technology, and banks and financial institutions, and in fact companies, service providers must acknowledge that, and must be able to provide other forms of payment approaches so that others who are not so adept can still participate in the economy," he said.
https://www.todayonline.com/singapore/dbs-citi-outages-prevented-25-million-payment-and-atm-transactions-being-completed-2298321
Piyush Gupta: Asia’s most disrupted digital banker
It’s a jarring moment. Piyush Gupta, CEO of DBS Group is often lauded as Asia’s top financier and has been tipped as a candidate to lead global rivals. The $62 billion Singaporean bank is also held up as a role model by Chinese insurer Ping An in its shareholder agitation at HSBC. Yet following repeated serious failures in its digital banking services, the Monetary Authority of Singapore has banned DBS from non-essential activities including M&A. It puts dividends at risk and will attract the eye of watchdogs overseas where the lender is busy expanding.
It undermines Gupta’s championing over the past decade of DBS as a technology company and adds to a monetary censure slapped on the bank. In May, it was ordered to retain a multiplier of 1.8 times to its risk weighted assets for operational risk, an increase from a multiplier of 1.5 times applied in February 2022. The higher figure translates to approximately S$1.6 billion, equivalent to nearly $1.2 billion in total additional regulatory capital, the regulator said at the time. Though the latest punishment is not financial, there are financial ramifications.
First, fixing the technology problems will require spending. What’s more, MAS has warned it could increase the capital requirements depending on DBS' so far uninspiring progress. Those factors will dampen investors’ expectations of a special dividend after its yearend earnings; the stock fell nearly 1% on Thursday. If dividends aren’t raised in line with Citi’s forecasts, analysts there reckon DBS is trading at 5.8% dividend yield for 2023 compared to compatriot OCBC’s visible path to 6.4%.
The bigger issue for the bank is reputational. The digital transformation smoothed DBS’ foray into large markets such as India, where the regulator in 2020 handpicked it to rescue a crisis-hit Lakshmi Vilas Bank, a local lender with over 500 branches. While that integration and the Singaporean bank’s separate purchase of Citi’s Taiwan consumer business last year are complete, regulators there are likely to pay closer attention to DBS. And whenever it is allowed to resume dealmaking, the bank may also find a cooler reception.
DBS Chairman Peter Seah warned that senior management would be held accountable for the tech lapses including on pay. On the back of record profits, Gupta earned $11.4 million last year, making him one of Asia’s highest remunerated bank CEOs. He told Breakingviews in September that he is more interested in becoming a naturalist than taking up another banking role whenever he leaves the lender he’s led since 2009. The shareholder returns Gupta has delivered remain enviable but his tech stripes look less impressive.
https://www.reuters.com/breakingviews/piyush-gupta-asias-most-disrupted-digital-banker-2023-11-02/
Service outages: MAS bars DBS from acquiring new business ventures, reducing branch and ATM network sizes for 6 months
• DBS has been barred from new business venture acquisitions for six months after repeated disruptions to its services in 2023 so far
• The Monetary Authority of Singapore said that the bank is not allowed to reduce the size of its branch and automated teller machines networks in Singapore
• DBS’ non-essential information technology changes have also been paused during this six-month period
• Its chairman Peter Seah said that the senior management will be held accountable for the disruptions and it will be reflected in their compensation
• MAS said that it will review the bank’s remediation efforts at the end of the stipulated period
SINGAPORE — DBS bank has been barred from acquiring new business ventures by the Monetary Authority of Singapore (MAS) and instructed not to reduce the size of its branch and automated teller machines (ATM) networks in Singapore.
In a media statement on Wednesday (Nov 1), the central bank said that during this stipulated six-month period, DBS is also required to pause its non-essential information technology changes.
The actions imposed by the authority were taken after repeated and prolonged disruptions of the bank’s services so far this year, and they are meant to ensure that the bank keeps “sharp focus on restoring the resilience of its digital banking services', MAS added.
On Oct 14, DBS customers experienced disruptions to banking and payment services.
They were unable to access many services provided by the bank, including the ATMs and fund transfers via online payment services PayNow and PayLah!, as well as payments at physical payment terminals such as those in stores and eateries.
DBS said then that the disruption was caused by an issue at a common data centre, Equinix, which was also used by other organisations.
Earlier in the year, DBS' customers were subject to disruptions in March and May.
After the March incident, MAS said that it directed DBS to engage an independent third-party to conduct a comprehensive review of the effectiveness and adequacy of the people, processes and technology supporting its digital banking services.
Several shortcomings were later identified, namely on the bank’s system resilience, incident management, change management and technology risk governance and oversight.
MAS said on Wednesday: “Following the independent review, DBS has set out a technology resiliency roadmap to address the shortcomings, improve system resilience, and better position the bank to meet future digital banking needs.
“The roadmap is being implemented in phases, with the changes affecting its system architecture design taking more time to complete.”
MAS, which reviewed DBS’ remediation plan under the roadmap, said that it was “satisfied with its scope” and the planned measures to improve system resilience.
It added: “In line with MAS’ expectations, DBS will hold senior management accountable for the lapses and the board will enhance its governance approach to oversee implementation of the roadmap.
“MAS has directed DBS to suspend all changes to the bank’s IT systems except for those related to security, regulatory compliance and risk management for a six-month period.
“This is to ensure that the bank dedicates the needed resources and attention to strengthen its technology risk management systems and controls. MAS will not approve any new business acquisitions by the bank during this period.”
ATM NETWORK AND BRANCH SIZES
The bank was also directed not to reduce the size of its branch and ATM networks to ensure that there are enough alternative channels for its customers in the event of further disruptions while the bank works to enhance the operational resilience of its digital channels.
This direction will be in force until MAS is satisfied with the progress of DBS’ remediation plan.
MAS said that it will review the bank’s remediation efforts after six months and it may then extend the duration of the measures, vary the additional capital requirement now imposed, or take further actions at that point.
“In the meantime, MAS will retain the multiplier of 1.8 times to DBS’ risk weighted assets for operational risk, which was imposed after the March and May 2023 incidents. “
DBS will take up to 24 months to put in place the planned structural changes to improve the resilience of its digital banking services, MAS said.
“In the meantime, it is possible that disruptions may still occur. In such situations, MAS expects DBS to promptly recover its services and communicate to its customers in a clear and timely manner,” it added.
MAS’ deputy managing director of financial supervision Ho Hern Shin said: “DBS must put in place immediate measures to ensure service reliability while it continues to invest in the longer-term efforts to bolster its operational resilience.
“We have imposed this six-month pause on the bank to give it the space to take the actions needed to maintain customer trust.
SENIOR MANAGEMENT WILL BE HELD ACCOUNTABLE
DBS'chairman Peter Seah issued an apology on Wednesday for the digital banking disruptions that happened this year.
He said: “The board apologises for the digital banking disruptions. When customers bank with us, they expect to be able to access our banking services conveniently, and at any time of the day.
“With the incidents of the past year, we have failed to live up to these expectations, and have also fallen short of our own standards.”
Mr Seah added that the senior management will be held accountable as an acknowledgment that the bank could have done better, adding that this will be reflected in their compensation.
DBS' chief executive officer Piyush Gupta said on Wednesday that the bank will be setting aside a special budget of S$80 million to enhance system resiliency.
"Over the years, DBS has focused on digital transformation so as to make banking simple, seamless and effortless," he added.
"However, we acknowledge that we must now do better to deliver on this, and are taking a multitude of actions across technology governance, people, leadership, systems and processes."
https://www.todayonline.com/singapore/dbs-barred-acquisitions-branch-atm-mas-senior-management-2295356
'This is not supposed to happen': Experts on DBS, Citi outage caused by data centre failure
Most banks have two or more data centres and redundancies at multiple levels if a primary data centre goes offline, industry experts say.
SINGAPORE: Banks are heavily reliant on data centres for their operations, making it crucial to have a backup data centre should one fail, banking and technology experts told CNA in the wake of a DBS and Citibank outage earlier this month.
These centres form the nerve centre of the banking ecosystem, and they store, manage and process massive amounts of data, the experts said.
In the case of DBS and Citibank, the data centre they used experienced technical issues with its cooling system during a planned upgrade, raising temperatures and affecting equipment. The incident disrupted both banks' payment and banking services on Oct 14.
Last Friday, the banks’ data centre provider Equinix said its initial investigation found that the incident was caused by a vendor contractor. The contractor had incorrectly sent a signal to close the valves from the chilled water buffer tanks, which affected the flow of chilled water to the cooling system.
Equinix said it is reviewing its processes and putting in place additional audits to prevent a similar incident during future upgrades.
The DBS and Citibank outage started on Oct 14 at 3pm and only fully resumed the following morning. Customers were unable to access both banks' apps and online banking or payment services such as PayLah! and PayNow. ATM services were also down at several locations.
Last Thursday, the Monetary Authority of Singapore (MAS) said it had ordered both banks to conduct a thorough investigation, noting that the downtime exceeded its limit of four hours within a 12-month period.
While MAS has no oversight over data centres, it expects banks to establish contractual agreements with data centre providers that incorporate its requirements on system availability, it said.
DATA CENTRES THE "NERVE CENTRE"
Data centres and their functions form the backbone of modern-day businesses, said Associate Professor Lee Poh Seng.
"At their core, they house a variety of components, including servers, storage systems, networking hardware like switches and routers, and the cabling infrastructure necessary for data and network communications,” said Assoc Prof Lee, who is from the National University of Singapore’s (NUS) Department of Mechanical Engineering.
Singapore Management University's (SMU) Dr Patrick Thng, who has close to three decades of experience in banking, including as managing director at various banks, likened data centres to a car's engine.
"From very fundamental office support applications … to banking critical operations, like core banking, credit card applications, ATM, foreign exchange trading, branch platform, digital banking services, (all rely on data centres)," said the director of a financial technology and analytics programme at SMU.
"Data centres have massive storage to store all your transactions and customer data. Banks rely on this data centre like a nerve centre."
COOLING SYSTEM CRITICAL TO DATA CENTRE
NUS’ Assoc Prof Lee pointed out that cooling systems are critical to data centres. Equinix uses a chilled water system, which works by circulating chilled water to absorb heat from the data centre environment and dissipate it outside the facility.
“They keep the hardware at optimal operating temperatures, and a failure could result in overheating and subsequent IT hardware failures,” he said.
According to the Singapore Computer Society, most data centres use centralised chiller plants that have built-in redundancy to enable continuous cooling in the event of equipment failure. Outages still occur, but are “very rare”, it said.
The recommended temperature range for data centres is 18 to 27 degrees Celsius, said Assoc Prof Lee, referring to guidelines by the American Society of Heating, Refrigerating and Air-Conditioning Engineers.
In Singapore’s tropical climate, with high year-round temperatures and humidity, robust cooling systems to prevent overheating are necessary.
When a cooling system is down and temperatures are not brought back to a safe range, overheating can ensue. This can cause intermittent hardware errors, data corruption, and in severe cases, permanent hardware damage, added Assoc Prof Lee, who holds two US patents in thermal systems.
Asked why it might have taken such a long time for the data centre to resume operations, Assoc Prof Lee said: “If not previously encountered, the incident's nature could present a learning curve, extending the recovery time. The adequacy and effectiveness of disaster recovery procedures in place could also affect the recovery time.”
BACKUPS, RECOVERY PLANS NEEDED
The importance of the data centre means that banks always have a backup data centre. Some banks also have two data centres that share the workload concurrently. In such cases, if one fails, the other can pick up the slack.
Dr Dennis Khoo, a managing partner at digital consultancy allDigitalFuture, said: "Generally for mission-critical applications like banks, there are multiple levels of redundancy.
"In most advanced banks, using the latest technologies, the database will be instantly replicated, that means they'll have a primary site and alternate site and the data is replicated instantly on both sites," said Dr Khoo.
The Singapore Computer Society said most data centres are designed and built with a certain level of redundancy and the ability to conduct maintenance in real time. They are also specifically built to meet the exact redundancy requirements of the business, it said.
A common uptime guarantee of a data centre would usually be at 99.982 per cent to mitigate possible disruption to its customers, the society added.
However, there is still 0.018 per cent of possible downtime. “Thus, the client must establish an efficient Business Continuity Management System and IT Disaster Recovery Plan to allow their critical IT systems and data information to immediately failover to the secondary data centre in the shortest possible time should such an incident occur.”
WHAT CAN BANKS DO WHEN DATA CENTRES ARE DOWN
That said, should all data centres fail, there is little – if any – service a bank can provide.
Dr Thng said banks activate what they call "offline mode", which means they render some services at branch offices using what is available. These transactions are then updated with the mainframe when the data centre is back up.
These services may include cash withdrawals – as banks have spare cash on hand – cash deposits, payment instructions and credit card transactions. In the case of DBS, it reopened branches to help customers with some services.
According to Dr Thng, IT incidents, such as a delay in processing, occur daily. Some of these go unnoticed by customers.
Still, service outages will negatively impact the bank’s reputation and may have some financial repercussions, for example, if a client incurs late fees from being unable to pay his bill on time due to an outage.
On reputational impact, Dr Khoo said banks would have “broken” their service commitments to provide around-the-clock service to customers.
“So definitely, in that sense, there will be some reputational damage in terms of your ability to serve customers properly. And with proper design, this is not supposed to happen.”
https://www.channelnewsasia.com/singapore/dbs-citibank-outage-data-centre-cooling-system-down-3861076
FUCK PIYUSH!!!!! FUCK PAP!!!!!
Two outages happening in less than a week, shame on ya!