In this episode Professor Steve Goldsmith interviews Zach Tumin and Maddy Want, co-authors of the new book "Precisely: Working with Precision Systems in a World of Data."
Professor Goldsmith talks with Zach Tumin and Maddy Want about their new book "Precisely: Working with Precision Systems in a World of Data." Want and Tumin review what a precision system is, explain the importance of real-time data, and discuss real-world examples. They review what precision looks like in practice - both the successes and failures - what we can learn from them.
Music credit: Summer-Man by Ketsa
About Data-Smart City Solutions
Housed at the Bloomberg Center for Cities at Harvard University, we work to catalyze the adoption of data projects on the local government level by serving as a central resource for cities interested in this emerging field. We highlight best practices, top innovators, and promising case studies while also connecting leading industry, academic, and government officials. Our research focus is the intersection of government and data, ranging from open data and predictive analytics to civic engagement technology. To learn more visit us online and follow us on Twitter.
Betsy Gardner:
Hi, this is Betsy Gardner, senior editor at the Harvard Kennedy School and producer of The Data-Smart City Pod. Since we started this podcast, we've had great support from our listeners, and to make sure that you don't miss an episode, please find us under the new Data-Smart City Pod channel wherever you listen. Make sure to subscribe so you get each episode, and thanks for listening.
Stephen Goldsmith:
This is Stephen Goldsmith, Professor of Urban Policy at the Bloomberg Center for Cities at Harvard Kennedy School. Welcome back to our podcast. We have two interesting guests today who have been involved in writing a new and about to be released and important book. I want to first introduce you to Zach Tumin, member of the faculty of Columbia University's School of International Public Affairs. Zach has been a senior fellow executive at the Harvard Kennedy School, worked at NYPD, been involved with Wall Street Financial Service Technology Consortium, New York City government, et cetera.
Welcome, Zach.
Zach Tumin:
Thank you Steve. It's great to be here.
Stephen Goldsmith:
And with him is his co-author, Madeline, and she is Vice President of Data at Fanatics Betting and Gaming. Previously was involved in product management for Audible and other companies, and they are authors of the upcoming book, Precisely: Working with Precision Systems in a World of Data.
So Zach, I've known you longer, so let me start with you. I read a lot of books and I thought this one was really, really good. Years ago, when I worked as deputy mayor for Mike Bloomberg, I tried to set up the city's first data analytics center, and I wouldn't say I barely succeeded. I'd say I was before that point, right? Made a little bit of progress, but not too much. The routines of government tended to overwhelm precision. So tell us a little bit what do you mean by precision and where did that word come from?
Zach Tumin:
I remember well over the years noticing that the routines of government overwhelm the possibilities for precision so often. And when we set out to do the book, Precision for us came to mean to create a specific change that managers wanted, where, when, and how they wanted it, nothing more and nothing less. And to do that using all the extraordinary data that was available to us today, big data and, as others have suggested, small data too, data of all kinds to fashion new approaches to old problems that had the unique and distinguishing feature of reducing the prospect of collateral damage of results that we didn't intend or wish. That seemed to us to be intriguing in its possibilities.
Maddy, you want to pick up from that at all?
Maddy Want:
I would only add we didn't make up the word precision, obviously, but we did invent this sort of customized usage of it and it's specifically to mean a combination of data, people, processes, et cetera. The point being it's the end-to-end operational system operating in pursuit of, for the purposes of precise change. And we use that to mean in contrast to legacy modes of operation, which perhaps were based on intuition, all based on trial and error, all based on experience. We're trying to distinguish that this new sort of more modern approach, not to say it hasn't been used in the past, but it's more visible more frequently now of really allowing data to drive and inform decisions and operating around that with the mentality of iteration, rather than an initial vision which must be executed at all costs. And we've seen it across a range of industries. You might think it's just a tech thing, but it's been everywhere.
Stephen Goldsmith:
I was struck by that word "journey," an interesting word. Give us another example of a journey. How do we think about journey in terms of personalization or customization around the needs of a resident?
Maddy Want:
One of the best things that we studied, the best examples that I liked the most was an organization who had developed a method to collect huge scale data about the movements of pedestrians in and around high capacity venues and arenas. And this was coming from citizens, people's cell phones that were recording back at just a very fine level of detail, but anonymously. And that was an important distinction at the time as well, the precise locations of people so that the company, the organization could analyze that at a massive scale and see how traffic moves in and around the arena in the case of a large event. And they use that for a couple of purposes.
One was traffic planning, unsurprisingly, deciding where parking lots should go. Another example was planning the emergency exits in the arena to try and predict where there would be crushes of traffic if an incident were to occur and people needed to escape quickly. And finally, it was to help customize the public transport plan around that venue to know where were people coming from and where did they go after they left the arena. And so that is a great example of using data to inform those strategies, rather than starting from the point of, "Well, what buses do we have and what do we think might be a good approach to this" sort of starting from the reverse end. That's one of my favorite examples.
Stephen Goldsmith:
Those are good examples.
Zach, how do you move from an example to a system, right? You guys talk about precision systems. What is a precision system as it relates to government and how do you get there?
Zach Tumin:
A precision system, as Maddy suggested, has unique and distinguishing feature of being an end-to-end arrangement of people and platforms to technologies, processes, politics that enable an initial problem statement that needs solving to explore and discover solutions and then move those solutions into action to realize outcomes of value that everyone would agree are essential. It's a system in the sense that there's a set of infrastructure, talents, and skills arrayed around it and they almost come together and work together and that turns out to be the task of leaders to define a problem clearly that can be solved, that would be valuable and important to do, and then see that through to the outcome. So that unlike just talking about AI, for example, and, "Hey, we need to just figure out all the fun things we can do with it," we want to put AI in the context of one of the tools that we use to move agencies and corporations to discover important opportunities and valuable outcomes.
And I'll say as well that the discovery is essential here and it's something that Maddy alluded to, which is that we don't take for granted the world as it's presented to us. We take for granted that we have tools to explore and discover the true nature of the world, to see the real patterns in the world, and to decide whether and how we like the future those patterns suggest to us. And if we don't, to fashion ways in which we can shift those patterns, change the trend to outcomes that truly do matter to us, and to do so in a way that don't cost us friends and enemies, disastrous results along the way. That then becomes the precision system that we are talking about, the end-to-end array and is the journey, Steve, that you mentioned earlier in our discussion. A journey starts with discovery, starts with pain points, goes through discovery, ends with some solutions, and then continues as we see these solutions in place, understand their impact, and refine the models that we use to predict what would happen next and what we should be doing next.
And so in that sense, it is a journey. Systems underlie it and people and technology and platforms make it possible.
Stephen Goldsmith:
So a joint question, Maddy to you and Zach, the relationship of AI to precision. When you talk in your book about a New York PD initiative Patternizer, Maddy, would you explain it and then, Zach, would you explain how you guard against being precisely biased as contrasted to precisely unbiased in the development? So Maddy, first you. What's Patternizer? How did AI play a role? And then Zach, what about the underside of this?
Maddy Want:
I would describe the relationship between AI and precision. Not all of precision is based upon AI. It's not a prerequisite. There are many ways to use data that are simple and heuristic and straightforward and still provide extremely precise results. And at multiple points during the book, we go on to tools like Microsoft Excel, the back of the napkin on which calculations are made just as valid precision tools. But, of course, a lot of the interesting discussions happen when we stop veering into AI.
And in the case of Patternizer, Patternizer was an initiative that the NYPD, under Zach, launched and invested in. And the goal of it was to be able to assess new crimes in the light of a library of historical crimes and quickly be able to compute which crimes are similar to this new crime that has just come in with the goal of being able to declare a pattern, to declare that a crime belongs to a pattern, that it is similar enough that it may have been the same perpetrators or that at least it could be investigated in a similar way.
And this is something that computers are very, very, very well suited to do. If given the same type of information about a massive variety of instances or events, they're very fast to pull out patterns and that was why the initiative got the name that it did. It was aiming to very quickly recognize patterns of robberies, murders, and grand theft and those were the three crime types that it attempted to categorize.
And the science behind it was interesting. The types of models and the approach that the data scientists use was interesting, but I think more interesting is sort of the governance aspect of the project and the steps they had to take to avoid bias and what happened next, once they had a relatively performant model and they actually did reach a good state of performance in terms of accuracy and predicting crimes that did eventually turn out to be connected in real life. Why isn't Patternizer now a household name? Well, it's because it went through a life cycle of adoption and eventual decline and failure that is interesting and, in my opinion, more interesting to inspect why that happened than it is to inspect a science, which is probably a good point to hand over to Zach.
Stephen Goldsmith:
All right, Zach, it's handed over to you and talk just a little bit about what Maddy handed to you and also how did you take steps to protect against inevitable discrimination in the application of the data?
Zach Tumin:
And I should say that when the deputy commissioner at NYPD under Bill Bratton, we initiated the discussions that ultimately led to the development of Patternizer, but we eventually left NYPD and the development of Patternizer took on its own life and thoughtfully so, I believe, in retrospect, well done. The issue was how to rinse out issues of race and gender and other potential bias from the application of Patternizer to incoming crimes. The answer was to eliminate the mentions of specific identifiers that would ascribe race in other potentially biasing aspects of the case to search for likeness and incoming crimes against the history of crimes in the past. And it was that, I think, purposeful and quite aware reworking of the traditional way of building predictive systems that I think helped the developers have confidence that, at least initially, they could claim that they had addressed some of the principle issues of bias that might be raised.
Ultimately, the problem for Patternizer was that it became rather constrained and never really fell into the detectives' workflows. It was not something that ran on the background sort of constantly standing. It had to be invoked and being invoked, it sat side by side other tools that could be invoked as well that were much more comfortably embraced as part of the ordinary detectives' workflows. It was also built to exclude juveniles from the database and rightly so. It managed only several crime types when there were many, many to go. And most importantly, its models were never really improved and it's important in predictive systems to constantly improve, refine models. In these cases, they weren't. And the original developers as well left. And so here was Patternizer, which was a good start out of the box that was homegrown and home developed by NYPD that nonetheless found this life sitting on a shelf, rather than in operations. And that is its story and its legacy.
Stephen Goldsmith:
So you make a compelling case about how much better cities could operate with precision. How would a city get there more comprehensively, rather than isolated instance? What would take it in terms of culture, organization, talent, capacity on the tech side? What would it be?
Maddy Want:
I think it's hard to overstate and I'll try and list some macro factors without diving too deep into any of them. First of all, the attraction of talent, of the type of skilled people who can do these types of analyses and who want to build these types of systems. It takes the acquisition and the retention of that kind of talent and so, working for the government in computer science capacities has to be more attractive than it currently is in comparison to the private sector where people with those skills, due to market demand, will be very highly compensated and can make immediate and lasting impact on whatever the corporation does. That's well known and well-trodden territory. I won't spend more time there.
I think one of the big issues that cities either are facing or will face if they try and transform themselves into sort of "smart cities" is needing to bring the general public along on the journey of understanding why this is happening to create a sense of trust and security in how the data is being used. There is obviously good reason to have suspicion about government collecting a person's data and what they'll do with it and how they'll use it. And there are successful instances of this around the world. We interviewed a great data science leader from Singapore and the cultural differences there were so stark because the level of trust in government to handle personal data is multiple percentage points higher than it is in general in the States.
Aside from identifying very specific narrow and delightful use cases of how data can transform city management like, for example, the traffic analysis around the venue and showing people this is what it can do and getting buy-in that way. I do think it's a journey of trust and vision that I can't imagine it being less than a multi-year, across-government initiative. I can't imagine a single leader making it happen. I can't imagine anything valuable happening within a three-year timeframe. It's got to be something that persists across election cycles and something that is generally understood to be a necessity of the future, rather than one wacky leader's specific vision.
Stephen Goldsmith:
Zach, finishing with you. You are a one-time wacky leader. How do you institutionalize instead of having it be just your idea?
Zach Tumin:
So these kinds of initiatives have to be supported at the highest levels of government. We notice that very often, as it's the case, that these initiatives are spawned at the lower levels of government. And so it's important to bring these initiatives forward to encourage innovation at the lowest level. Ultimately, what matters here is the governance of these systems. Trust of citizens in this kind of computing is essential and it's as difficult as it could possibly be. It takes a tremendous amount of willingness to create transparency to the black box. Some people can understand the models, but the fact is that these models can be very difficult to understand and even transparency to professionals is not always forthcoming. The challenge here is to develop these, as I think the developers of Patternizer recognized and did, how important it is to anticipate where the issues are going to be that will undermine trust and begin to bring them forward.
Stephen Goldsmith:
So most important question of the day. When will the book be published?
Zach Tumin:
Precisely is available now for pre-purchase. Will be formally released May 23, 2023.
Stephen Goldsmith:
Perfect. Well, this is Stephen Goldsmith, Professor at Harvard talking to two esteemed guests, Zach and Maddy, about to be released book. Must reading for every government person, I think. The book is Precisely: Working with Precision Systems in a World of Data. Thank you very much for your time today.
Maddy Want:
Thank you, Steve.
Zach Tumin:
Thank you. Thank you Steve.
Betsy Gardner:
If you like this podcast, please visit us at datasmartcities.org or follow us @DataSmartCities on Twitter. And remember to subscribe at the new Data-Smart City Pod channel on Spotify, Apple Podcasts, or wherever you listen. This podcast was produced by me, Betsy Gardner, and hosted by Professor Steve Goldsmith. We're proud to be the central resource for cities interested in the intersection of government, data, and innovation. Thanks for listening.