Data-Driven Governance: Rick Cole & Stephen Goldsmith Opine at CityLab 2014

Stephen Goldsmith: At the Harvard Kennedy School, we invite the chiefs of staff of the largest 30 cities to come together. A few weeks ago, Ed Glaeser came before them. He said that you really don’t know what works until you do a random controlled experiment. There was a collective sigh: “You’ve got to be kidding! Where are we going to get $2 million to do a RCT?” Today’s conversation is: How do you know what works when you don’t have $2 million and you want to test the waters?

Rick, Los Angeles has a “back to basics” mayor and the deputy mayors are focused on improving operations. But what are you doing about someone like the assistant manager at the Department of Public Works? Does he or she have the same culture of performance that you do? How do you change his or her environment? How are you managing that change in culture?

Rick Cole: “Patiently” and “systematically” are the bywords that we stress. Changing a culture takes time.  Change has to be systematic. This mayor’s office aims to treat the 44,000 people that work for the City with “trust” and “empowerment” because they are the ones who ultimately will create the culture we seek. 

Our budgeting process is shifting toward a focus on performance. We emphasize that it’s a marathon, not a sprint. Eventually, we’ll shift people away from saying, “When I get back the staff I had in 2007, I will be able to do the job I was doing in 2007.” We’ll move them forward to, “How am I going to use the resources I have now to do a better job in the year ahead?” 

We’re implementing CompStat, which is the world-class model used by our police department. People come from all over the world to see how the LAPD uses data, statistics, and evidence to improve outcomes in police work. It’s made us the safest big city in the country.  On the basis of per capita crime, we are safer than we have been since 1949. CompStat works. But no one from the other 34 departments had thought about crossing the street and paying attention to that model—until Mayor Garcetti took office last July. 

So our approach to emphasizing performance is systematic: implementing performance budgeting, implementing CompStat in all 35 departments, creating a culture that promotes training and innovation. That means it’s not going to happen overnight. 

Cities launch lots of great initiatives that make for a good sound bite in Governing Magazine—but five years later, the cool idea isn’t in practice and things haven’t changed. We want to build a sustainable culture. It has to be systematic.

Stephen Goldsmith: When I was Deputy Mayor of New York, I spoke with heads of each of the wastewater treatment plants (there are 12 or 13). I said: “What are we doing that makes your job worse?” One said, “The absence of revolving funds. On July 31, we have to buy as many parts as we can because we know we can’t buy them for the next three months. If you want us to use our professional skills, you’ll change the way you allocate the revolving funds and procurement.”

What are the one or two biggest structural obstacles in LA government preventing the data and culture reaching the person who sees the problems but can’t solve them, because the structure won’t allow it?

Rick Cole: I don’t think absence of revolving funds is our biggest problem. I think it’s lack of training. The City of LA has a $7.2-billion budget and 44,000 employees. Last year, before Mayor Garcetti took office, the citywide training budget was $0. That’s a big impediment, because we talk to people about what we want done but too often they have no idea how to do it. 

The second biggest challenge is communications. We have four different email systems. The mayor can’t even send an email to all 44,000 people without having to go to three other people to send it out. 

Do our systems suck? They absolutely do. Our IT systems suck and our personnel systems suck—no question. But those are actually fixable by smart people who are passionate about public service. First, we have to develop improved understanding through better communication and expand our capacity through increased training.

It’s not the quality of our people that’s hurting performance. It’s the quality of our systems.

I think in the day-to-day it has less to do with some blinding breakthrough that says: “This program is utterly worthless and this program is doing a fantastic job – let’s cut the funding for the first and double the funding for the second.” It has to do with a lot of programs that are not paying attention to how they could be better. They’re evidenced based—based on evidence from 20 years ago, 50 years ago, or even 100 years ago. The model we use for deploying firefighters in the state of California is absolutely evidence-based. It’s based upon evidence from an era when cities were built out of wood that didn’t have fire sprinklers, back when 100 percent of our calls for service involved fighting fires. Now, less than 19 percent are fire calls. 

We’re operating off really old evidence. We aren’t using new data to redesign a better deployment system. In moving to a data-driven culture for government, no one is talking about getting rid of the fire department. But I would certainly redeploy them based on the FireStat system that we’re setting up, based on the model of LAPD’s CompStat.

Stephen Goldsmith: What do you do when, for example, Northwest LA transportation department evidence shows that it’s outperforming Southwest LA? You’re using the data, and you see that units of performance over cost of performance in one quadrant is off from that in another quadrant. What do you do as the manager? How do you hold those folks accountable and allocate the resources around you?

Rick Cole: In the real world, we’re seeing that for the first time in our planning department, where the department leadership now has an understanding of how each individual planner is processing case load. There are certainly good reasons why someone might take longer on a particular case, but, for the first time we actually have an individual scoreboard on how people are doing at their jobs. 

What we’ve learned in the last 15 years with CompStat is that if you take a punitive approach to performance data —“You guys are screwing up,” or “You individuals are not doing your job”— people will stop telling the truth. That’s not very helpful. 

But if instead you say, “What’s holding you back? Is it the lack of equipment? The lack of training?”—that becomes a Socratic use of Socrata data (Socrata is the company that manages the City’s performance data portal). That’s where CompStat really provides benefits. It’s not the numbers—its what you do with the numbers. 

When you see that North Valley and South LA are performing differently, you ask: why? What is one doing right that can be applied to the other? What could be improved in both? If the numbers don’t change after two or three months and if the excuse is basically “My dog ate my homework,” then pretty soon it’s clear from the numbers that there is a performance problem—not a lack of equipment, training, or best practices. It gets outed in the CompStat process if you’re not paying attention to what the data is telling you and not doing a good job.

Audience Question: Don’t you think there has been a paradigmatic shift from the traditional method of evaluation? For instance, in France, our government has launched 40 evaluations in old ministries, but nobody reads the reports in the end. In our experience, the traditional methods of evaluation are boring, and not desirable because they are more data-centric than human-centric. The people running projects are not part of the evaluation. Don’t we need to think about a more desirable way to prove results? I think we could shift from scoring people to engaging in the Socratic conversations you mentioned. What would be your advice for more desirable methods of evaluation? 

Rick Cole: Anything worth doing can also be overdone. That seems to be the problem with data. Anything that makes sense in balance, if you make a religion out of it, ends up being abused. 

Using data is an art and a science. You have to bring both. I agree with you: It’s not about the numbers, but about what you do with the numbers. Having “neutral” people with green eyeshades come to judge and second-guess frontline staff is not particularly helpful. 

Data that doesn’t get utilized, that isn’t embraced, and that isn’t understood is not particularly helpful data. If we want to actually improve performance and use data to solve problems, we need partners who embrace it, rather than outsiders or supervisors imposing it.

I think the next frontier is comparing data across cities. It is one thing to compare our data to past performance. It’s another thing to look at a much wider dataset from similar cities. When we begin to meaningfully access how we’re doing in comparison to other cities, I think that’s going to be the revolution. Clearly that’s created a revolution in education. But we are still in the “every city left behind” category. 

When we begin to have data across cities, then Oakland will know whether it’s beating Milwaukee because we’ll have scoreboards! Right now, we think we’re beating our past performance. But we don’t have comparisons. The fact that we’ve got a platform that a lot of us are using is beginning to generate some consistency.

Audience Question: Where have you seen the use of data put together really effectively? 

Rick Cole: The LA Police Department. They have been consistent, they’ve told a story, they’ve stuck to the story, they’ve learned from the process, and they’ve produced extraordinary results. 

Stephen Goldsmith: This CityLab audience really cares about evidence, but they don’t have a lot of money for random controlled trials. How do you think about evidence, short of the gold standard of an MDRC evaluation?

Rick Cole: We in government don’t do research. Colleges and universities do. Almost every one of your cities has one nearby. Professors, students, and grad students are curious themselves. We’ve done a lot of partnering with colleges.From the beginning, when we decided to implement CompStat in 34 departments, we sat down with the smartest people from USC who focus on this. They’re now doing, with foundation funding, a longitudinal study of how this is working. We’re getting feedback in real time. It can be intimidating to think, “Where are we going to get the money for  this incredibly complicated and expensive study?” Go to your colleges and universities and ask them which of their professors do this. 

Stephen Goldsmith: Rick, do you have a closing comment?

Rick Cole: I don’t want to make an artificial distinction, but I think it’s useful to understand that there is evidentiary data and there is performance data. I would define evidentiary data as: You don’t really know what the solution is, so you’re looking in the raw data to gain a better grasp of the problem as the basis for solving the problem. Performance data comes in when you have a pretty good idea of what works, and you want to see how well your activities are producing the outcomes you’re seeking. It’s not a black and white line between the two. But I think it’s important not to mix them up as if all data were the same.