The way in which in-house legal departments assess, select and evaluate external counsel continues to bear remnants of the Stone Age, according to some observers.
“Most in-house departments don’t think of using metrics or are too lazy to try them and even when metrics are available many in-house department tend to ignore them,” says Lexpert columnist Richard Stock of Catalyst Consulting in Toronto. “So the process of selecting external counsel turns out to be rudimentary in 90 per cent of the cases. That doesn’t mean that the counsel selected aren’t good counsel, but that the process for selecting them and structuring the arrangements is frequently not very business-like.”
But, like everything else in the profession, that’s changing. It’s becoming harder and harder for legal departments and law firms to ignore the useful data that modern technology can extract from the outputs that everyday operations provide. TD Bank Group, for example, has found that data metrics – the science of uncovering hidden patterns from raw data – can play a significant role in choosing panels of approved law firms.
“We discovered that we can leverage a great deal of internal data to help us with our decisions,” says Natasa Milojevic, Senior Manager, External Legal Services with TD in Toronto.
The difficulty lies in navigating the data. “The profession is still working at that,” Milojevic adds. “There’s tons of data but the problem is how to wield it, how to use it to make meaningful decisions and how to make it resonate with the people who are actually retaining the law firms.”
Milojevic maintains there’s a “disconnect” in large in-house departments between the teams that come up with the metrics, typically the legal operations teams, and the decision makers.
Stock echoes this. “Relationship-based selection is still the way most folks pick counsel,” he says. “The people in legal procurement love the metrics, but lawyers have a cultural overlay that militates against widgetizing legal services procurement.”
The challenges may be even greater for the 50 per cent of law departments in Canada and the United States that engage four or fewer lawyers.
“It’s tough to shout metrics at small law departments, apart from figuring out what they’re paying their lawyers, because metrics only make sense when you have the volume and type of data that lends itself to sensible analysis,” says Rees Morrison, a New Jersey-based consultant with Altman Weil, Inc.
And while there may be plenty of data out there, its reliability can be questionable. “I find that the data available to make dependable or truly informed decisions is very weak in the legal market,” says Friedrich Blase, Global Director of Pangea3, a Thomson Reuters legal outsourcing company based in New York.
Morrison has also surveyed the data collection practices of 130 law firms and law departments. “Most respondents are methodologically unsound both in collecting data and in how they portray it,” he says.
Still, because data metrics are making headway generally, they are also making headway as a tool for choosing and evaluating counsel. While, as Stock says, counsel selection may still be a largely relationship-driven exercise, the process is bound to change: because expertise is displacing reputation as a driver in the choice of law firms, information is bound to become more and more crucial.
Indeed, the Blickstein Group’s 8th Annual Law Department Operations Survey found that 56 per cent of law departments surveyed had some sort of formal metrics program in place in 2015, up from 34 per cent in 2014. Although the programs were not necessarily directed to choosing external counsel, there’s little doubt about the direction legal departments are headed.
On the technology side, companies like Lex Machina, Sky Analytics and Serengeti are aggressively collecting benchmarking and industry data to support their programs. On the user side, automobile manufacturer GM and the Marsh & McLennan Companies, risk managers and insurance brokers, are two companies that have led the way in using data metrics as part of the counsel selection process.
“Metrics have really only taken hold in the last couple of years, but since then there’s been significant growth,” says Bennett Borden of Drinker Biddle & Reath LLP in Washington, DC, where he is not only a partner and litigator but also the firm’s Chief Data Scientist, responsible for Drinker Biddle’s data analytics strategy. “Clients are looking for more and more of that type of information.”
A similar trend is emerging in Canada. “Clients are definitely starting to use metrics to select counsel and manage our performance,” says Judith McKay of McCarthy Tétrault LLP in Toronto. “What they measure is what matters to them, but we use that information for our own benefit as well.”
Increasingly, the information measured is subjective in nature. “We’ve been scored on things like how responsive we are, the quality of our work product and the extent to which we employ strategic thinking,” McKay says. “These are things that are hard to measure with specific data but they can be scored numerically.”
On the quantitative side, clients may be interested in the number of alternative fee arrangements being offered, the extent of the law firm’s experience with AFA, the nature of value-added services, the firm’s overall diversity profile as well as the profile for the team working with the particular client, and how well the firm has conformed to and communicated about budgets.
As it turns out, the hype around Big Data has been around for years. But 2015 may have been the year that the hype actualized. Drinker Biddle and Littler Mendelson P.C. are just two of several American law firms that established in-house data analytics practices in 2015.
Human capital, Borden argues, does not in itself allow a law firm to do work at a price clients want to pay. “They need qualitative empirical information in order to make the right decision about putting together the service and the service delivery team,” he maintains.
Besides, the benefits of data analytics are not limited to efficiency and cost-oriented information. “Big Data can be huge in case analysis, e-discovery and in helping clients leverage information to get the results they desire,” he says.
Lincoln Financial Group, based in Fort Wayne, Indiana, has gone so far as to use metrics to calculate win rates, not to be confused with performance evaluations. The company measured “wins” by combining settlement costs with legal costs on similar high-volume matters. It discovered that the results combined with the legal costs varied markedly among external counsel. Consequently, it has started moving business to the firms with the best “win” record.
Big Data fluency, it appears, is making its mark. It may not be long, it seems, before metrically challenged firms find that metrics are barriers to entry with lucrative clients. Clearly, Drinker Biddle is betting that the firm’s jumpstart in this arena will prove to be a major market differentiator.
“Establishing the chief data scientist position is an internal and external signal that we’re a firm that is going to take advantage of analytics and data,” Borden says. “Many companies have called us out of the blue simply because of this and the way that we’re using it. My bet is that data officers will eventually become a component of every law firm’s DNA.”
Several years ago, TD embarked on a significant RFP. With the support of TD’s internal procurement teams, Milojevic and her team figured out a way to analyze hourly pricing, get baselines, assess a qualitative score and then compare that score against qualitative metrics.
“On the qualitative side, we measured things to which we could assign a number, like the extent of our previous experience with a firm, the geographic coverage a firm provided, and the depth of staffing in certain areas and at various levels of seniority,” she says.
TD then quartiled the firms based on price as well as their quantitative and qualitative scores. “We generally dropped the firms in the bottom quartile of each practice area,” she says. “And when we found firms that were high on quality and high on price, we asked ourselves whether that combination was appropriate for the kind of work we were seeking.”
Although the metrics were significant in choosing TD’s panels, lawyers in charge of individual matters were free to choose the approved firm that they want to use on any given matter. “We are now looking at ways to leverage metrics to assist our lawyers in selecting from the approved panels,” Milojevic says.
But too much granularity can have its disadvantages. “It’s easy to miss the big picture,” says Rick Kathuria, National Director, Project Management and Legal Logistics at Gowling WLG in Toronto. “When you examine time metrics, for example, you can see how many hours a firm has spent on internal meetings, which clients don’t like to pay for, but what is really significant is the overall cost.”
Too many clients, Kathuria adds, focus on minutiae rather than examining average transactional or case costs. “The best value from metrics is the overview they can provide.”
Metrics are also turning up as tools for in-house departments to manage existing relationships.
TD has designed a relationship management framework that has a qualitative scorecard at its core. At the end of each matter, the in-house lawyers score the firm they used for responsiveness, substantive knowledge, strategic advice, accurate budget-setting and proactive budget updating, innovativeness, meeting deadlines, proactive communication and value delivered for costs.
TD also designates some six firms in Canada and eight firms in the US as “strategic” firms. These firms are drawn from within each panel and across the entire enterprise and are subject to a more robust performance review.
“We meet with the strategic firms semi-annually, we go over the spend and how it translates across practice areas, then compare that to the value-added services we’ve received and how we’ve leveraged them,” Milojevic says. “All of that helps the conversation about how and where the firms can do more for us.”
But it’s not just about doing more for the client. The process benefits the external firms because it can create opportunities for new and better work.
“Let’s say a firm wants to break into commercial lending,” Milojevic says. “With the data we’ve collected, we may be able to tell them that they have to do more in certain areas, like the value-added services space, before they can get that work.”
For the most part, however, the data that’s being used by legal departments is generated internally by the departments themselves or by law firms retained by in-house counsel. What’s been missing is the general availability of external data to fuel counsel selection.
But that’s changing. Lex Machina has so far made the biggest splash. The firm uses an extensive database of information derived from regulators, court rulings and other data sources to inform a statistical analysis, including strength, weakness and performance assessments of all the lawyers, parties and judges involved in federal intellectual property litigation. In other words, in-house counsel involved in this kind of litigation now have access to external data about a law firm’s experience with specific types of disputes and a data-based account of how they fared with various judges, opposing parties and opposing counsel.
Owen Byrd, General Counsel at Lex Machina in Menlo Park, California, cites the case of a deputy GC for IP at a large tech company who wanted alternatives to the large law firms that had previously handled the company’s litigation. Using legal analytic software, he discovered 50 boutique firms that had never acted against his company, had experience in the company’s technologies and had recent success in court.
Lex Machina’s legal analytics platform uses natural language processing and machine learning to derive patterns and insight from millions of bits of information. In-house counsel can use the results not only to choose and evaluate counsel but also to explain their choice to management and internal clients. The IP project has been so successful that Lex Machina is now moving into the antitrust and other areas.
Byrd believes it’s just a matter of time before legal analytics become even more sophisticated. “Someday savvy corporate counsel will be able to correlate litigation data with billing data to assess the effectiveness and efficiency of outside counsel, which will provide new ways to benchmark them against other firms,” Byrd writes in Corporate Counsel.
In Canada, the situation is different.
“The US has a lot more information that’s easy to query and glean,” says Kathuria, who sees his role at BLG as akin to that of chief data scientist. “But having said that, there are some databases in Canada that have useful information that just needs to be mined.”
At least one Canadian start-up is going through the available databases with the aim of starting up a legal analytics subscription service. But it’s early days domestically and it could be a while before Big Data, and certainly external Big Data, makes it to the mainstream of the Canadian legal market.
Given the pace in the US, however, things may spill over the border more quickly than anticipated. In a review of US developments, Legaltech News recently called 2015 the year that Big Data became “part of our everyday vernacular.”
Arguably, an even stronger indication that Big Data is looming is the emergence of a New York City Band that calls itself “Big Data.” It turns out that the band has a lot in common with lawyers. The group describes itself as a “paranoid electronic music project from the Internet, formed out of a general distrust for technology and The Cloud (despite a growing dependence).”
Their biggest hit song is “Dangerous.”
Metrics That Matter: What to measure for outside counsel performance
- Per cent of matters for which full-year budget was submitted on time
- Per cent of matters managed for which forecast updates were submitted on time
- Actual spending versus budget, by matter
- Average blended rate for all law firm attorneys who billed to the client by matter and across all matters
- Success in predicting total cost-resolution range for a matter
- Other process goals, including timely completion and submission of reports, early case assessments, mock trials, after-action reviews and lessons learned