Wall Street to Web StreetA Report on the Online Brokerage Industry
The commercialization of the Internet is quite possibly the most significant business development of the 20th century. Not only has the Internet given birth to an entirely new ecommerce industry, but it has also revolutionized the way traditional brickandmortar companies conduct their businesses. Swept up in this stream, many traditional brokerage firms have begun providing online brokerage services, while numerous cyberbrokerage firms have also sprung up to capitalize on the growing demand for online trading.
As Credit Suisse First Boston observed in its analysis of the online trading environment, the 4th quarter of 1998 was a record quarter for the online industry, then the 1st quarter of 1999 was quite simply a complete blowout. Surprising almost everyone in the industry (including most capacity planners at online trading firms), online trading volume surged to an alltime record level of 499,476 trades/day in Q1 99.1 As a result of this unexpected surge in trading volumes, a number of online brokerage firms witnessed peak capacity utilization rates much higher than they had planned for, in many cases causing performance degradation and outages. As a consequence of these outages, in January 1999 our office began receiving a heavy influx of consumer complaints about various online brokerage firms.
Pursuant to my authority under New Yorks securities and consumer protection statutes, I initiated an inquiry of numerous firms in the online brokerage industry. As part of this inquiry, my office reviewed over 120,000 pages of documents, spoke with online brokerage firm representatives and experts regarding information systems and online trading practices, and visited firms at twelve sites in five different states. On November 22, 1999, after 9 months of factfinding, document review and site visits, we issued an indepth report detailing our findings, which fall into four broad categories.2
First, we learned that a number of online brokerage firms were unable to meet the demands placed on their systems at various points in 1999, when online trading hit unexpectedly high volumes. In examining why these performance problems occurred, we also learned that assessing the adequacy of a firms information systems capacity is a complicated undertaking because there is no single number or metric that can accurately represent capacity.
Second, we identified vulnerabilities in each tier of the brokerage firms information systems, all of which contributed to the repeated performance degradations and outages that occurred last year.
Third, we assessed the responsiveness of the firms customer supporta critical fallback option when systems suffer delays or go downand concluded that in some instances this service has been wholly inadequate.
Finally, our investigation revealed a number of ways that online brokerage firms could make more thorough disclosures regarding issues that may affect customer access and order execution.
How Online Trading Works
As a prelude to discussing our findings, it may be useful to summarize what happens when a customer places an order with an online broker. Online brokerage firms typically utilize a threetier architecture consisting of a frontend system, middleware and a backend system. The first tier, or frontend system, generally comprises computer servers or Webservers. The second tier, or middleware, provides messaging, routing and access to the firms trading system. The middleware determines the type of request the user is placing (such as a request for research, quotes or customer support), and routes that request to the appropriate part of the system for a response.
Trading functions occur within the third tier, or backend system. Once a customer enters an order, the frontend system will return a message to the customer asking for order confirmation. At that time, the customer is usually provided with a realtime quote for the relevant security. Once the order is confirmed, it passes from the frontend to the backend of the brokers system. During that time, the order undergoes a vetting procedure to check, among other things, for restrictions on the customers account, to verify that the customer has adequate capital, and to ascertain whether the customer is authorized to trade on margin. This process typically takes place without human intervention, according to preprogrammed instructions.3 If the order does not pass the preprogrammed vetting procedures, the order exception will be manually reviewed by a broker, afterwhich the order willgenerally be sent to themarket for execution.
Once an order has been executed, the market maker, exchange or ECN forwards an execution report back to the online brokerage firm through the firms back office provider or clearing agent. The agent or provider then forwards the execution notification to the firms backend system, which is used to update the customers account information and order status screens.
One of the most striking conclusions we reached during our inquiry is that assessing a brokerage firms system capacity is a complex matter. Each firms system has many components, any one or combination of which can become a bottleneck, constraining a firms overall ability to meet user demand. For example, if virtually every component of a firms system operates at a low percentage of its available capacity, the systems overall operational capacity will still be limited by the component with the lowest capacity threshold. For that reason, in measuring a firms capacity, the focus should not be on the overall system, but rather on the weakest link in that system.4
Furthermore, the percentage of capacity utilized at a given firm varies widely throughout the trading day, spiking dramatically before and during market open and again before market close. The capacity sufficient to accommodate ordinary trading volume may not be adequate during these narrow time frames of high utilization, although on average, the demand on a system for the entire day will appear manageable. Similarly, capacity can be stressed during times of high volatility, as well as when there is a large concentration of trading activity in a small group of securities.5
Constraints on Brokerage Systems Technological Capacity and Performance
Bearing in mind the complexity involved in assessing a systems capacity, we learned that there are a number of macro issues that can have a significant impact on the overall performance of an online brokerage system.
Reliance on Older Mainframe Systems
While the architecture of every system is unique, the hardware used to house an online brokerage firms backend trading system generally consists of either a mainframe system or a bank of computer servers. However, some online brokerage firms or their service bureaus use mainframe systems that appear to suffer from limitations that may affect their ability to offer uninterrupted service.
Most significantly, mainframe systems were never designed to run on a 24 hoursaday, 7 daysaweek basis.6 They were made for backoffice functions, not for highvolume, online transactions that involve a direct interface with consumers. At certain points early in 1999, these legacy mainframes appear to have strained to accommodate the extremely high demands of the online trading environment.7
Furthermore, mainframe computers cannot service customers around the clock because, by their very nature, these systems require planned downtimes and outages. Every mainframe must be taken down at least some time every day in order to update the database to reflect the days transactions, to print reports, and to perform numerous system housekeeping functions.
Inability to Expand
System scalability, or ability to expand, may also impact the overall performance of an online brokerage firms system. Scalability in turn will be affected by both the design of the system and by various realworld physical constraints.
For example, one firms critical trading functions were being run off a single server. This configuration eventually reached the scalability limit of the firms central processing unit, or CPU. At that point, the firm could no longer add more CPU capacity and undertook a number of alternative measures to\ alleviate the workload on this critical component, including offloading some functions to other components of the firms backoffice system. With the installation of a software upgrade, however, some of the offloaded work shifted back onto the critical component, causing the components CPU utilization to jump to full capacity on several days. The CPU saturation caused the system to suffer multiple outages. The firm eventually overcame the problem by configuring a second critical component identical to the first, effectively doubling its capacity to handle the trading process. Nevertheless, the firms primary trading system and database may still present scalability and reliability limitations.
Physical limitations posed by a firms technology facility can also affect scalability. In short, as firms add more customers and more hardware, they ultimately exceed the space allotted to house and operate their equipment. The adequacy of available electricity and cooling may also limit expansion.
Inadequate Capacity Planning
A firms overall capacity planning and management will have a significant effect on system performance. As detailed in our report, a number of online brokerage firms have engaged in aggressive advertising campaigns in order to garner a piece of the burgeoning online investment market. This advertising, combined with public enthusiasm for Internet stocks, prompted an increase in customer accounts and total trading volumes that surpassed the firms expectations, causing strain on its technical capacity.
As a result of sharply increasing demand, a number of firms implemented hardware and software upgrades onthefly, often making changes without rigorously examining their potential impact on other parts of the system. In fact, even with these onthefly f ixes, some firms found it difficult to add components quickly enough to keep up with increasing demand, while others had to postpone previously planned hardware upgrades in order to concentrate on handling the existing demand.
Problems with Individual System Tiers
In addition to these overarching constraints, we also identified specific problems with regard to each tier of the online brokerage system architecture.
FrontEnd Web Interface
Virtually all of the online brokerage firms that we looked at experienced some type of problem with their frontend systems in 1999. These problems included slow customer response times and the involuntary loggingout of customers. As detailed in our report, many of these difficulties apparently resulted from capacityrelated problems. In fact, one firms Webservers appeared to have been operating close to or at 100% capacity virtually every day in January and February of 1999.
One of the most prevalent difficulties experienced by online brokage customers earlier in 1999 was an inability to log in to customer accounts. Generally, in order to place trades or check account information, a customer must log in to a firms member areas using an account number and password. This number and password are verified by the system before allowing access to sensitive customeronly areasa process referred to as customer authentication. We found a number of instances where customers were able to log in to a brokerage firms Web site, but were not able to access the restricted member areas. Some of these problems occurred because of delays in customer authentication, which appeared to stem from capacity bottlenecks in the frontend system.
For example, one firm that suffered capacity bottlenecks on the server database in which it housed its customer information saw a slowdown in the authentication process. As customer volumes increased, the databases ability to service all login requests degraded because authentication requests were competing with other demands placed on the database. Eventually, these demands prevented the database from providing authentication information in an acceptable time period, causing delays in the customer login process and occasionally causing customers to be timed out before they could finish logging in.
Another firm experienced problems due to the capacity constraints on a critical database that generated what is known as alert messages, which are messages regarding information that the customer has requested or that the firm believes the customer should receive.8 The firms system was designed to check for alerts during the login process, pull up any active alerts, and present them to the customer. If the system was unable to retrieve alerts, the customer could not log in. During early 1999, the firm utilized only a single database server for alerts. When demand surged, that sole server could not keep up with all of the requests. Since the customer login procedure could not be completed until these alerts were retrieved, numerous customers faced login difficulties.
An online brokerage firms middleware is responsible for providing the integrated messaging between the front and backends of the firms threetier system. Generally, this routing of messages is handled by software. In essence, the middleware is a critical facilitator through which all information must pass. Just as many firms frontend systems have suffered from capacityrelated problems, so has the middleware that connects the front and backends of the systems.
In addition to capacityrelated problems, a firms middleware can also be adversely affected by software problems. For example, one firms customers suffered a number of delays due to problems with messaging software connecting the firms backend system and its Web trading servers. As a result, customers experienced slow response times in Web trading and were unable to access certain account and transaction information over a period of a week.
A brokerage firms backend is probably the most critical part of its system. It is there that trading and order routing logic resides, where the vetting of trades occurs, and where customer accounts are maintained and updated.
A major component of most firms backend systems is a mainframe computer, or a bank of servers, which handles the firms order vetting and routing functions. Through our investigation we learned that during times of high volatility, many firms were operating their primary trading system CPU at or above capacity limitations.9 We also learned that in some instances, capacity limitations on the backoffice provider systems, as well as bottlenecks in the lines between the brokerage firms and these providers, appeared to adversely affect timely processing and execution of trade orders.
Probably one of the greatest limiting factors affecting a firms backend system involves the database used to maintain customer and transaction information. Regardless of whether a firm handles its backend tasks internally or through outsourcing, every backend system is reliant on the customer database. Heavy demands on these databases appeared to be a frequent source of bottlenecks and often affected order routing and the reporting back of filled orders.
For example, one firm experienced bottlenecks and reduced response time when the number of transactions per day significantly exceeded the maximum number of transactions that the database server was designed to handle. In another case, a backoffice provider stored all pending open orders in what is known as an open order file. Generally, the open order file is broken down securitybysecurity, with each security being allocated a certain amount of file space based upon historical norms, including market volatility and volume concentration. During a heavy concentration of trading in a small number of securities, the backoffice provider began to suffer severe capacity bottlenecks as the amount of storage space devoted to each of these heavily traded securities hit preset ceilings. When these limits were reached, an extensive amount of input/output activity occurred as the system continually tried to find additional storage space. The lack of storage space, coupled with this excessive input/output activity, created a capacity bottleneck, which strained the backoffice providers orderprocessing capability, threatened the brokerage firms ability to accept orders in those securities from its clients, and ultimately led to a number of outages.10
As a longterm solution, the brokerage firm undertook a project to move the open order file to internal memory, which would result in improved speed and reliability. In the short term, however, the firm was forced to take more immediate steps to avoid bottlenecks and outages. To this end, the firm implemented a policy termed blocking. When securities are blocked, online investors orders for those securities are rejected by the brokers Web site, and customers are directed to call the firms branch office and place the order over the phone, thereby slowing the order flow.11
We also discovered that some firms encountered delays in the reporting of trades and the updating of customer accounts to reflect executions. One cause of these delays might be system or hardware failures by the market makers or exchanges to which these orders are routed. Delays might also occur as a result of deficiencies in the communication lines connecting the market makers and exchanges to the brokerage firms or their backoffice providers. These lines can conduct only a finite amount of information at any given time, but as order volume increases, the information that must flow back and forth also increases. In addition, some online brokerage firms (or their backoffice providers) may impose restrictions that inhibit the efficient use of line capacity; at least one of the market makers we spoke with had, but could not use, available lines that would have enabled it to report trades more expeditiously.12
Another factor affecting the speed with which order executions are reported to investors is the speed at which brokerage firms and service bureaus can transmit information back and forth to each other. For example, while the bandwidth of the lines connecting one service bureau to a market maker can carry data at speeds up to 56,000 bits per second, the service bureau itself can only transmit the information over those lines at 4,800 bits per second, or less than 10% of the total capacity of those lines.13
Reporting delays might also arise due to system slowdowns and outages at the brokerage firm itself. Interestingly, one firm designed its system so that, before it needed to engage in any intervention to offset stress, the system would first attempt to permit its queue of customer execution reports to build. In essence, the process of reporting filled trades back to customers was given a lower priority than order entry.
In addition to suffering technologyrelated capacity deficiencies, some brokerage firms did not appear to provide adequate customer support services. Our review of the customer service records of a number of online brokerage firms confirmed that, at times, customers were unable to access telephone support. This was especially true during times of high volume and volatility.
An analysis of key telephone statistics conducted by one firm revealed that in late January1999, the total number of calls answered within 60 seconds dropped from almost 100% to approximately 25%; the number of calls answered within 120 seconds dropped from almost 100% to 33%. During the same period, average response time climbed to around 10 minutes, peaking at about 13 minutes in the first half of February. Although response times subsequently improved, they again rose to 9 minutes in early May 1999. At the same firm, the number of abandoned calls also increased. Almost half of all customer calls were abandoned by the customer during the period from midJanuary to midFebruary. Even more striking, on one day in early 1999, the number of calls abandoned exceeded the number of calls answered by almost onethird.
Disclosures of the Risks and Limitations of Online Trading
Another area of focus for our inquiry was the disclosures made by online brokerage firms regarding known system limitations. As detailed in our report, an expectation gap appears to have developed between what consumers expect and what online brokerage firms can truly provide. There is clearly a need for a concerted effort to correct these erroneous expectations. To that end, additional disclosures may be warranted regarding certain risks and limitations of online trading.
Messages Acknowledging Order Entry
Generally, there appears to be a great deal of confusion about the messages that investors receive during the order entry process. Traditionally, when trades could only be placed through a broker, there was never really a question about the information provided to investors at the time of order placement; either the broker would confirm the execution when the order was called in, or would call the customer back once execution took place. With the advent of online trading, however, these expectations appear to be changing. If online investors are unsure of the status of their orders, they may expose themselves to unnecessary risk by placing duplicative orders, or failing to take timely action to cover themselves during a market turnaround.
To prevent this problem, additional regulations regarding disclosures during the order entry process should be considered. In the interim, firms should endeavor to improve their disclosures. It serves neither the firms nor the investors interest to have an expectation gap that could potentially result in investor dissatisfaction.
Disclosure Regarding Known Problems with Order Entry
Even firms that utilize backoffice providers to perform their order processing functions are not immune to tradingrelated capacity problems. Yet despite having known, and in some cases recurring, problems, brokerage firms do not disclose these problems to their customers.
For example, we learned that bottlenecks at one firms backoffice provider occasionally caused order processing delays that required the brokerage firm to disable a feature that allowed for automatic submission of rudimentary customer orders to the marketplace for execution. Turning off this function required a manual release of each order to the backoffice provider, thereby slowing down the order routing process. The firm never advised its investors that it had a recurring practice of disabling this function.
Similarly, as discussed previously, the backoffice provider of one firm had trouble adding new incoming orders to its open order file. As a result, the firm blocked certain heavilytraded securities to slow the order flow and give the backoffice providers system extra time. Nonetheless, the broker never disclosed that it intentionally shunted orders for select securities to a manual telephone system, despite the fact that the problem existed for approximately six months.
Another firm often intentionally reduced the number of orders that could be transmitted at any given time to prevent the system from exceeding its capacity. During times of high volatility, online investors encountered difficulties in entering orders and the firm experienced delays in reporting trades. Yet despite these known system limitations, the firm never disclosed them to online investors.
Disclosure of Outages and Slowdowns as They Occur
Despite some of the wellpublicized outages that the major online brokerage firms suffered in 1999, as a general matter there is still inadequate disclosure when firms experience an outage.14 For example, while my office was visiting the technology center of one of the largest online brokerage firms, that firm suffered an outage which lasted for approximately 45 minutes. During that time, other Attorney General personnel attempted to log in to the firms Web site in order to ascertain whether or not the outage was disclosed. After searching through a number of the firms Web pages, we were unable to find any mention of the problem. When we raised this concern with the firm, we were informed that outage information was not posted on the home page, but should an online investor have logged into an account to place an order, the investor would allegedly have been informed that trading functionality was inoperative. During subsequent outages at other firms, we found similar failures to disclose in a prominent location.
Online brokerage firms could ensure that investors learn of outages in a timely manner by posting notices on their home pages promptly when outages occur. Handinhand with disclosures of system outages, online brokerage firms should also be required to provide disclosures when they encounter system bottlenecks that impact their ability to process orders, route them for execution, or report executions back to customers. This holds especially true if these delays are known to the firm and due to chronic system problems.
Disclosure of Outdated Account Information
Many online brokerage firms necessarily run substantial nightly batch programs to update customer accounts to reflect the most recent transactions. With one possible exception, all of the firms that rely on nightly batch processing presently lack the ability to provide real time updates of account buying power, available margin, and positions. As a result, the intraday information in an online investors account is often based upon the previous days close. Moreover, if batch processing takes longer than the allotted time on a given evening, an online investors account balances could be outdated by as much as two days. Since this delay in updates is a known limitation, brokerage firms should disclose the constraint to investors. Additionally, investors should be informed that they may be unable to access their account information while batch processing of updates is in progress.
On a related matter, because most firms cannot update account buying power and margin obligations on an intraday basis, online investors were in some instances able to place orders that exceeded the actual buying power of their accounts at the time. They may also have placed orders that breached other limitations, such as preset margin spending limits. These problems have created unexpected investor indebtednessoften in substantial amounts.15
This problem is all the more troubling when one observes the potential impact on selfdirected IRA accounts. Since online brokerage firms utilize account information that could potentially be outdated by a day or more, customers can, and indeed have, made purchases that far exceeded the buying power of their IRA accounts, requiring them to forward additional moniesmonies they never intended to invest, which are not tax deductible, and which they may not be able to withdraw for many years without penalty.
Insufficient Disclosures Regarding Disabled Features
In one instance, a brokerage firm asked its market maker to refuse to accept cancellations or modifications of market orders from its online investors during market hours unless the firm actually contacted the market maker to approve the change. Yet, despite the fact that cancellation and modification features were disabled for all of the firms online investors, the options were never removed from the order entry screen, and the firm apparently never disclosed the limitation. In the absence of such disclosures, investors could have mistakenly relied on their attempts to submit amodification or cancellation without knowing that the functionality had been deactivated.
Trading by Touch Tone Phone Not Available During System Down Times
Many online brokerage firms offer investors the choice of using a touch tone phone as an alternative to trading over the Internet. Depending on the architecture of the firms system and the system of its clearing agent, however, touchtone trades and Internet trades may both be routed through the same system for execution. Thus, an outage that prevents the execution of Internet orders would have the same effect on touchtone orders. If a brokerage firms touch tone trading suffers from this limitation, it should be disclosed.
Disclaimers by Firms
Many online brokerage firms generally disclaim responsibility in their customer agreements for a laundry list of system problems, including their own hardware and software problems. Since the online brokerage firm is generally in the best position to assess its own systems, it is questionable whether these disclaimers are appropriate. This is especially true in light of the inadequate disclosures already discussed.
Although I have spent a great deal of this article discussing some of the shortcomings in online trading technology and deficiencies in brokerage firms disclosure, I do not mean to imply that investors should avoid trading online.16 Indeed, in the course of our inquiry, we discovered that many of the specific technical deficiencies that led to slowdowns and outages earlier in 1999 have already been remedied. But, our dialogue with the industry also revealed that the risk management procedures utilized by online brokerage firms needs to be strengthened to avoid a recurrence of similar problems in the future. Additionally, these firms must remain vigilant against permitting their customer account volume to exceed their overall system capacity.
The susceptibility of mission critical information systems to temporary failure, and the concomitant losses that might ensue from these failures, has elevated the industrys awareness of the need to adopt and implement risk assessment and mitigation strategies. This process is particularly important for online brokerage firms and other financial institutions, where customers demand extraordinarily high performance because delays may prevent them from taking advantage of market conditions or expose them to market losses.
Recommendations to the Industry
As a first step toward adopting better risk assessment and mitigation strategies, I offer several recommendations that I hope will facilitate the responsible growth of online investing.
First, online brokerage firms must tell the truth about their technology and services. At present, online customers have only limited information about such matters. Overall, brokerage firms should make a concerted effort to provide better disclosures regarding system limitations. They should disclose slowdowns or outages as they occur, in a prominent and easily accessible location on their home page. Firms also should notify customers of heightened margin requirements, limitations on customer account buying power, and order cancellation procedures. Furthermore, investors should be informed of market conditions, such as extreme volatility, that might affect their trading practices.
Second, the public should be able to understand and compare the service quality offered by various online brokerage firms. I believe that this goal is best accomplished by mandating disclosure standards that allow for applestoapples comparisons of technology performance, customer services, and technology development processes, preferably through a combination of standardized selfreporting and independent thirdparty reviews. Firms should assess and improve their own internal quality assurance processes, and consider using models widely recognized in the computer industry, such as CMM or ISO 9000, to fully assess their software development processes.
Third, while we must be careful not to place regulatory handcuffs on the online brokerage industry, the SEC and selfregulatory bodies, such as the NYSE and NASD, should consider several specific and narrowlytailored changes. In particular, these entities should consider adopting requirements regarding the documentation and retention of system outage information, system performance standards, and customer service data. Regulatory bodies might also consider whether it is appropriate to require an effective and accountable process for online brokerage capacity planning; to provide further guidance regarding disclosure of online brokerage information system capacity and reliability; and to mandate disclosure of online brokerage spending on information systems technology.
To further advance the discourse on these issues, my office will be sponsoring a number of roundtable forum discussions over the coming year. It is my sincere hope that these round tables, which will be open to the general public, will help to continue the dialogue begun with the issuance of our report, as well as the report of SEC Commissioner Laura Unger.
One of the most lasting impressions I have taken
away from our inquiry into the online brokerage industry is how fast online
trading has grown in a short amount of time. During this period, many
online brokerage firms have made substantial strides in overcoming technology
and system limitations. Yet that work is by no means complete. Through
continued discourse and disclosure, coupled with the implementation of
responsible risk assessment practices, we can work together to move this
burgeoning industry into the 21st century, and beyond. At the same time,
my office will continue to work with and monitor the industry to ensure
that the goals I have set forth above are met, and that investors and
markets are adequately informed and protected.
All contents © 2000 Glasser LegalWorks. Click here for reprint permissions, credits and disclaimers.