Case Studies : ROYAL BANK OF CANADA’S SOFTWARE WOES

Royal Bank of Canada

Founded in 1864 and chartered in 1869 as the Merchants’ Bank of Halifax, Royal Bank of Canada (RBC) took its current name in 1901. In 1941, based on its total assets of $1 billion, RBC became Canada’s largest bank. Twenty years later, RBC installed its first computer, making it the first bank in Canada to employ such technology. In October 2003, the organization had 60,000 employees, totaled $413 billion in assets, and served 12 million customers. Including worldwide operations, RBC has spent $1.6 billion on technology.

On Monday, May 31, 2004, the RBC information technology staff made a program- ming upgrade to what has been described as “key banking software.” According to Martin Lippert, RBC’s vice chairman and CIO, a glitch in the bank’s computer systems “was most likely caused by a single worker entering ‘a relatively small number’ of incorrect pieces of code during the update.” The glitch resulted in a massive computer failure that affected millions of banking customers around the country. Lippert speci- fied that the incorrect code was related to “transit numbers or field identifiers in a table.” The mistakes worked their way through the bank’s systems quickly, but con- spicuously. RBC had already discovered the problem by six o’clock the next morning. Unfortunately, recognition of the error wasn’t even half the battle.

In fact, RBC fixed the programming error that had been made on May 31 early on Tuesday, June 1. Still, several days later, millions of RBC customers were still unable  to check their account balances, had not received their paychecks, or had automatic payments or transfers delayed. The reactions of customers, as could be expected, were tinged with displeasure. Some customers merely suffered the inconvenience of not hav- ing access to information that they were accustomed to having at their fingertips all of the time. Other customers, those who count on their paychecks to get by, had their  lives affected more seriously. One such person, a law firm executive assistant named Andrea Mitchell, was forced to ask her employer for a cash advance to make up for   her temporarily lost paycheck and to meet her personal expenses.

So, if the human programming error was corrected so swiftly, why did RBC contin- ue to experience problems related to the glitch over the course of a work week? The glitch put into motion a chain of management and control procedures that exacerbated the problem. Procedures that were intended to fix the problem instead caused a logjam of activity from which the bank could not recover immediately.

The first question that most people asked was, Why weren’t backup systems used to keep business flowing while the main systems were being fixed? As stated on the bank’s Web site, “Back up facilities exist in case our primary facility is disabled. As a matter of policy, therefore, all program changes are implemented simultaneously to both the primary and backup facilities.” Therefore, the same error that compromised the main system also affected the backup system.

According to Lippert, in addition to the programming code update being entered incorrectly, the new code was not tested properly before it was deployed. RBC policy calls for all new pieces of software to be tested thoroughly before they are used in pro- duction systems.

On Tuesday, June 1, with the knowledge that random duplications of transactions were occurring in its systems, RBC decided to suspend its end-of-day processing   rather than let it run with incomplete or inaccurate data. When verifying the health of its systems took longer than expected, RBC was left with a backlog of transactions to process. All the while, new transactions were pouring in, as they would on a normal business day, and these added to the logjam. On Wednesday, June 2, the IT department was confident that it could make up the ground lost on Tuesday and remain current  with Wednesday’s transactions. The newer transactions were placed at the end of queue because of a sequential numbering system, so the backlog had to be cleared before RBC could process any new transactions. Also slowing down the effort was the bank’s decision to monitor data processing manually following the glitch to make sure that no coding errors persisted. Trying to process two days of transactions at once proved to be more difficult than the bank imagined it would be.

As if the event itself were not bad enough for business, RBC came under criticism for the way in which it handled the public relations end of the problem. First, RBC made assurance that all systems and accounts would be normalized by Thursday, June 3. Confident in this assessment, the bank’s CEO, Gordon Nixon, left for Europe on Wednesday, June 2. When customers were still experiencing difficulties with their accounts later in the week, the bank appeared weak by not having its leader available  to address customers’ concerns. As a possible reason for Nixon’s untimely exit from the scene, John Layne, of the crisis management company Contingency Management Consultants, surmises the common organizational flaw in which lowerlevel employees are loathe to pass bad news up the corporate ladder. In Nixon’s absence, other RBC executives took turns addressing the media and the public, though the first public com- ments on the glitch did not come forth until late Wednesday afternoon. Having a single representative of the bank to interface with the public would have inspired more confi- dence in customers.

Unfortunately for RBC, the cumulative effect of its errors was widespread. About 62,000 government workers in Ontario and 10,000 in New Brunswick didn’t receive their automatic deposits even if they didn’t do their personal banking at RBC. Their provincial governments did use RBC to route the payroll, so the deposits didn’t reach the workers’ own banks. Hundreds of thousands of other workers around Canada ran into similar delays or difficulties accessing their deposits. The government of Ontario indicated that it would issue physical checks to employees who still had not received their deposits by the Monday following the initial computer glitch.

RBC’s service disruption had one particularly dangerous side effect, over which the bank had little control. Opportunistic scam artists seized on the glitch to unleash a phishing scam that targeted RBC customers, taking advantage of the customers’ frus- trations, security fears, or simply their naïveté. An e-mail message with the subject line “Official information from RBC Royal Bank” stated that “due to the increased fraudu- lent activity within our site we are undertaking a review of member accounts.” The email linked to a page that looked like RBC’s official Web site that instructed cus- tomers “be sure to enter both client card number or business card number & password otherwise your account will not be verified and your access to the account will be blocked.” Once RBC customers entered the requested information at that bogus site, hackers could access their accounts.

RBC announced that it had resumed normal business practices during the week beginning June 8, 2004. The bank continued to resolve discrepancies from service charges and overdraft interest that resulted from the disruption and indicated that all accounts should be reflected accurately in customers’ next statements, by June 30. On June 18, RBC announced that it had hired Crawford Adjusters Canada as its “adminis- trator for non-banking-related costs and losses in the bank’s processing disruption.” The bank made claim forms available, by phone and on the Internet and set a deadline of September 30, 2004 for customers to file claims. RBC was to handle all claims under $100 itself, leaving larger claims to Crawford. The total loss to RBC was contin- gent on the number of claims filed. However, that cost could rise significantly based on a classaction suit filed in Quebec that requested damages of $500 for each customer that was affected.

In hindsight, analysts agree that RBC handled the technology portion of its problem effectively. The bank erred mostly in its estimates of how long the recovery period would be. In addition to issuing a formal apology, RBC created an area of its Web site that was devoted to detailed explanations of the problem and updates on the progress  of restoring its records. The bank also enlisted IBM Corporation as a consultant to investigate the causes of the problem and provide guidance on how to avoid such prob- lems going forward. RBC offered refunds to its customers for any banking fees that they incurred as a result of the computer problems and agreed to reimburse other finan- cial institutions and their customers for certain expenses that may have resulted from RBC’s errors. These moves could go a long way toward restoring faith in customers of the bank, who, in the immediate aftermath, were understandably concerned about the security of their finances. One client said that she didn’t intend to leave RBC just then, but, “If it happened again, I may not be so loyal.”

Sources: Ian Austen, “Bank Software Woes Leave Many Canadians Without  Money,” New York Times, June 7, 2004; “RBC Admits Human Error, Scam Artists Act,” Ottawa Business Journal, June 10, 2004; Lindsay Bruce, “RBC Glitch Still Not Ironed Out,” ITWorldCanada.com, June 7, 2004; Chris Conrath, “Anatomy of a Snafu,” ITWorldCanada.com, June 11, 2004; Paul Waldie, “RBC Calls in Help,” Globe and Mail, www.theglobeandmail.com, accessed June 19, 2004; Mel Duvall, “RBC’s Account Imbalance,” Baseline Magazine, July 1, 2004; and CP, “RBC Still Making Up for Glitch,” London Free Press, www.canoe.ca, accessed June 19, 2004.

Case Study Questions

Provide a summary of the security and control problems experienced by the Royal Bank of Canada. How did this problem impact the organization and its clients?

What management, organization, and technology factors contributed to the con- trol problems at RBC?

What was the most critical flaw in the security and control systems in place at RBC? Explain your answer.

How well did RBC respond to its computer problem? What do you think the bank might have done differently to prevent the problem and to handle it once it did occur?

If you were a manager at RBC, how would you prevent a problem like this from happening?