IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

A big thanks to all the folks who’ve donated to IPA’s anti-poverty work before the year end (you can also donate through Dean Karlan’s Facebook fundraiser through tomorrow, credit to his brave daughter on that one.)

  • Thirteen prominent economists offer their favorite econ papers of the year, but the paper making a splash this week is from Melissa Dell and Pablo Querubin, comparing two approaches to combatting insurgency during the Vietnam War. They exploit the fact that the U.S. Army and Marines had authority over strategy in different parts of the country and took different approaches. While the Army used an overwhelming bombing campaign, the Marines took a “hearts and minds” approach, embedding troops in villages for security and conducting development projects. They conclude the bombing increased the insurgency and worsened attitudes towards the U.S., compared to the hearts and minds approach. Or as someone commented, it turns out you can’t bomb somebody into liking you.
    • Their method for finding similar villages (that were either targeted for bombing or not) to compare got a standing ovation at an NBER talk:

[Newly] Declassified Air Force histories document that one of the factors used in allocating weekly preplanned bombing missions was hamlet security (Project CHECO 1969). A Bayesian algorithm combined data from 169 questions on security, political, and economic characteristics into a single hamlet security rating. The output ranged continuously from 1 to 5 but was rounded to the nearest whole number before being printed from the mainframe computer.

The study estimates the causal impacts of overwhelming firepower by comparing places just below and above the rounding thresholds, with being below the threshold used as an instrument for bombing.

  • While many people have advocated for framing predictions about the future in terms of probabilities, David Leonhardt uses a Kahneman story to explain that most people have difficulty thinking probabilistically, and usually just round up or down to a yes or no. Instead he suggests that stories explaining why things might plausibly turn out one way or the other could work better. Similarly, earlier this year Robert Shiller wrote that economists, lost in numbers, have failed to appreciate the human instinct for storytelling. He went through newspapers before and after the great depression to gauge sentiment at the time, and concludes that economists rarely account for the power of people’s mental narratives to drive economic cycles. But he suggests this is an area where economists might learn something from the humanities.
  • I mentioned a few weeks ago that the U.N. was sending a special rapporteur on extreme poverty and human rights on a tour examining extreme poverty in the U.S. His observations on terrible conditions for the poor in the U.S. were pretty stunning. But he also concludes that one reason for America’s lack of action in the face of such suffering, or even for implementing policies that make poverty worse, is the stories other Americans tell themselves about the poor:

He found that stereotypes serve to undermine the poor — and are used to justify not coming to their aid. “So the rich are industrious, entrepreneurial, patriotic and the drivers of economic success. The poor, on the other hand, are wasters, losers and scammers,” Alston told NPR. As a result, he says, many people believe that “money spent on welfare is money down the drain. Money devoted to the rich is a sound investment.”
He spoke to politicians and political appointees who were “completely sold on the narrative of such scammers sitting on comfortable sofas, watching color TVs, while surfing on their smartphones, all paid for by welfare.”
But Alston says he met people working full time at chain stores who needed food stamps because they couldn’t survive on their wages.

  • In his job market paper NYU’s Kevin Munger describes a randomized trial where he used twitter bots/fake accounts to remind uncivil people to be nicer in political discourse, and it worked. So maybe there’s hope for humanity?
  • Scientists, you’ve been caught. Adding new information to Wikipedia entries led to that new information being cited more in the scientific literature.
    • It also makes me look with more interest at the trial Chris Blattman did at the request of Wikipedia, using his course to have students write or improve Wikipedia articles.
  • One of the better cryptocurrency efforts I’ve seen: you can donate your extra CPU cycles to mine cryptocurrency for the Bronx Bail Fund. (Caveat: I haven’t tried or vetted it yet, and yes I know about the electricity), h/t Colin Rust.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

MotorcyclesCanoeSurveyor Bench

(Research team in Sierra Leone, photo credit: Jeff Steinberg)

Please support the poverty researchers who really work amazingly hard (I can promise you HQ is not spending it on reliable copiers). Poverty-action.org/donate.

  • Some nice news this week, the MacArthur Foundation awarded its big “100&Change” 100 million dollar big idea award to the International Rescue Committee and Sesame Workshop. They’ll use it to implement an evidence-based “toxic stress” reduction and education program for Syrian refugee children in four countries.
  • A Nobel Prize winning biologist created a fund to help women scientists hire assistance for domestic work that disproportionately falls on women, like childcare and cleaning.
  • The Development Impact bloggers pick their favorite papers of the year and offer short summaries.
  • Tavneet Suri’s reflections on a great first six months for Vox Dev (wow, has it been only 6 mo?!) and some of her favorite reading.
  • For anybody who wants a quick Dev Econ 101, Jess Hoel had her students tweetstorm some classic papers. Follow the links in this thread:

  • Her reflections on how it went here. She thinks the skills it takes to put a paper into tweets – thinking about the audience and presenting the core of an idea briefly and visually – are very generalizable.
  • Job: IPA’s hiring a qualified senior research associate in Mexico to study good policing practices (Dec 31 deadline!)
  • Students of color, particularly African-Americans, make up a lower share of Econ Ph.D. students and faculty than other disciplines. Peter Blair Henry, dean of NYU school of business offers a fellowship for promising minority undergraduates to spend two years in New York doing research and being mentored in preparation for applying to Ph.D. programs. (Deadline Feb 16th.)
  • Interview advice for econ job candidates:
  • And if you’re going on a trip, remember our IPA 2017 Great Holiday Travel Podcast Playlist (and leave your additions there in the comments)

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

Screen Shot 2017-12-15 at 12.34.41 PM

  • First a word from my sponsor – IPA’s kind enough to let me use some time writing these links up almost every week for the last 2.5 years, but there’s no such thing as a free link. If you’d go to www.poverty-action.org/donate and help us make our end-of-year budget I’d appreciate it.
    • And, I’ll draw up a few winners (at random of course) for your choice of a) a tote bag from Ghana (long story), b) a very cheap lunch with some of our staff in New York or c) My 84-page doc of links I didn’t have room to post. Let us know your preference in the donation form comments box.
  • Via Claudia Sahm, a great interview from the Minneapolis Fed with Princeton economist Anne Case (also scroll back to April & May in the Financial Times Alphachat podcast to hear a great interview (iTunes) and interesting bonus chat (iTunes) about her career and reacting to blogs critiquing of her work).
  • We don’t often hear the stories behind the data that goes into papers, but the enumerators in the field work really hard. Here’s one story about an enumerator tracking down a participant from one of Chris’ studies in Uganda, 9 years later.
  • I recently mentioned profs might want to check their letters of recommendation for  gender biased phrasing (here). A tech company is doing something similar for job posting language:

    Textio found certain phrases such as “disciplined” and “tackle,” used more often by Netflix and Google, respectively, statistically correlated to a more male-dominated applicant pool. Netflix didn’t respond to a request for comment and Google declined to comment.

    Atlassian Corp. , a maker of workplace collaboration tools and a Textio client, said that after it overhauled the language in its job postings, women accounted for 57% of the class of new-graduate hires working in engineering, product management and design in 2017, compared with 10% two years ago before the language changes.*

  • In the wake of the increasing revelations of #MeToo in academia, the Women In  Economics at Berkeley blog interviewed four men figuring out how to be better allies to their classmates and colleagues.
  • Most development aid doesn’t go to the neediest parts of countries, Ryan Briggs writes. Aid projects tend to cluster in the better-off and urban areas, which may be simply because it’s harder to get to the more rural impoverished places that need it more.
  • Psychologists are trying to solve the replicability problem with a collaboration of 183 labs on six continents, who’ll volunteer to run the same study simultaneously (not all labs will run all studies).
  • People’s nominations for worst or weirdest cryptocurrency promotions. (h/t David Batcheck)

PS – related to the ask above, you can also use our Amazon Smile link for shopping, to donate a small portion to us (the Smile Always chrome extension will remember to redirect you there).


And, from Reddit/DataIsBeautiful, lighting strikes follow the path of shipping lanes (exhaust from ships increases likelihood and intensity of thunder storms). (h/t Max Galka)

Lighting Strike over shipping lanes

* Yes I realize it’s a before-and-after story.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • An interesting new Center for Global Development working paper looks at the effects of family planning services becoming available across Malaysia. They find that it helped girls earn more later in life, even those too young to benefit directly from the program. Girls just born grew up to earn more (and were more likely to have their elderly parents move in with them), presumably through a generalized improvement in women’s empowerment in the area (h/t David Batcheck).
  • Lucia Diaz-Martin, Rachel Glennerster, and Ariella Park warn that using whether women work or not as a proxy for empowerment in research can be problematic, because it matters why they’re working
  • It’s never a good omen for a body of literature when David Roodman takes his microscope to the code. In this case, just in time for the holidays, he releases a Worm Wars prequel. GiveWell rates deworming as one of the most effective charitable causes, but the case is based on a just few data sets. One is Hoyt Bleakley’s historical finding from the 1800s that hookworm eradication in the American South resulted in improved education. Roodman went back to re-examine the case with the benefit of a 100x larger sample, thanks to more recently added census data. He now thinks the improvements in education observed might have been a continuation of a trend that had already been in progress.
    • Despite his findings, GiveWell still thinks the case for current deworming efforts in developing countries is strong and continues its recommendations in support.
  • But in a remarkable development, hookworm is back in the American South.
  • Some in government are pushing for policy to be made based on “sound science.” FiveThirtyEight explains this is being used as a Trojan horse, turning the scientific process against itself. Historically this tactic has been used by tobacco and energy industries to delay policy until scientists have conclusively proven something or under the guise of needing more rigorous studies to establish certainty. Conveniently, it turns out that topics like smoking and climate science can’t be subjected to randomized controlled trials.
  • Nature asked several statisticians how to improve science. Opinions varied on if “to p or not to p” is the problem or it’s more deeply embedded in norms of how statistics are used in different fields.


And, in 1988 The Lancet asked clinical trial authors to do a subgroup analysis to find out for which patients (out of their 17,000) had benefitted most from using aspirin to prevent heart attacks:

Peto, a statistical rigorist, refused — such analyses would inevitably lead to artifactual conclusions — but the editors persisted, declining to advance the paper otherwise. Peto sent the paper back, but with a prank buried inside. The clinical subgroups were there, as requested — but he had inserted an additional one: “The patients were subdivided into 12 … groups according to their medieval astrological birth signs.” When the tongue-in-cheek zodiac subgroups were analyzed, Geminis and Libras were found to have no benefit from aspirin, but the drug “produced halving of risk if you were born under Capricorn.” Peto now insisted that the “astrological subgroups” also be included in the paper — in part to serve as a moral lesson for posterity.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • GiveWell’s latest charity recommendations add a new one, “No Lean Season,” which helps hungry rural farmers (so far in Bangladesh) get to cities to find their own employment while they wait for their crops to come in. You can hear the audio of Yale’s Mushfiq Mobarak describing how it went from an idea to one of the most cost-effective charities here.
  • It’s “best of” time of the year:
  • Jobs!: IPA, J-PAL and a bunch of affiliated orgs post jobs here. Evidence Action has a bunch of jobs posted here (including directing the aforementioned No Lean Season).
  • It’s also letter of recommendation season, check yours for gender-biased phrasing here or with the newer version here.
  • A bizarre story and warning for academics going to Denmark. An American professor working at a University there is being charged criminally for giving an invited talk to Parliament and tax authorities about her research studying offshore tax havens. Immigration officers have charged Sociologist Brooke Harrington and 14 other academics sharing research with working outside their permits, but it gets better:

    Ironically, on the day Harrington learned of her criminal charges, she was notified that she’d received an award for research dissemination from the Danish Society for Education and Business.

  • How two CNN journalists went undercover in Libya to expose that migrants being turned back by the EU were being sold as slaves there. (The EU has known of the human rights abuses there for a long time).
  • Last year the New Yorker reported that mentally ill prison inmates in Florida are regularly tortured, sometimes to death.
  • Compare that to Chicago – America’s largest mental health facility turns out to be Cook County Jail (which has 9000 inmates! That’s 30 percent bigger than the average college). So the Cook County Sheriff appointed clinical psychologist Nneka Jones Tapia as warden. She then began offering treatments to inmates who needed it and the jail is now working with researchers to test a series of programs designed to ease transitions after release.
  • Mugabe reportedly offered to trade his wife to stay in power. More on the details of his final days in office. But the best Mugabe story is that in the year 2000 he won the national lottery.
  • How the fight for a soda tax in Colombia got ugly (and John Oliver on how cigarette companies are trying to intimidate developing countries who are considering warning labels on cigarettes, h/t Colin Rust).

And if you get frustrated by how long it takes your experimental machine learning code to run, the original creators of the Oregon Trail first person adventure game did it without access to a computer.  Using teletype access to a mainframe they did stuff like adjust probability of snow:

The snow probability in the game was defaulted to zero during miles 0 to 950 of the trip. Bill and Paul developed a parabolic formula that, when graphed, produced an upright U-shaped curve covering miles 950 to 2000, with mileage along the X-axis. During each turn through that segment of the trip, the computer generated a random number that was compared to the curve. If the random number fell under the curve, as it often did at the curve’s beginning and end, snow occurred, representing the two mountain ranges. If the number fell above the curve, as it often did during the mileage between the ranges, snow was less likely. Using different curve formulas, this technique was used for several of the game’s variables to produce different event probabilities as a function of trail location.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


We’re putting up the links early this week for your travel enjoyment.

  • If you’re traveling, end your trip smarter than you started! We’ve posted the IPA 2017 Great Holiday Travel Podcast Playlist with podcast feeds and specific episodes we liked.
    • It’s got stories from around the world, research podcasts, and, in preparation for the holidays, three different episodes on how to disagree constructively. So feel free to just play those over the Bluetooth speaker while you’re cooking with the family.
  • And this Thanksgiving, economists might be watching you at dinner (and in your sleep). A paper seems to have been taken down (cached version here), showing that families from politically different voting districts have shorter Thanksgiving dinners together. Here’s how they figured it out:

Location tracking data comes from Safegraph, a company that aggregates location information from numerous smartphone apps. The data consist of “pings”, each of which identify the location (latitude and longitude) of a particular smartphone at a moment in time. Safegraph tracks the location of more than 10 million Americans’ smartphones, and our core analysis focusses on the more than 17 trillion pings Safegraph collected in the continental United States in November of 2016…

Home locations are determined by looking at where each person in our sample is most frequently between 1 and 4am. … This procedure identifies the home location of over 5 million people in the November Safegraph sample, and we link these locations with their corresponding voting precinct, two-party vote share, and census demographics using GIS software.

With Mugabe finally leaving (maybe?), this ad might have to be retired.


IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • On Wednesday the Nigerian women’s bobsled team became the country’s first team to qualify for the winter Olympics, and will be the first African team to compete in the Olympic bobsled event.
  • Experts are arguing about what to call what’s going on in Zimbabwe – whether there’s a waiting period before declaring a coup (the African Union frowns on coups apparently), a bloodless coup, or maybe “protective coup” (where the leader is kept safe). Given the importance of the first lady to understanding the political situation, I would have gone with “coup de Grace.” (But follow Kim Yi Dionne & Naunihal Singh for actual information.)
  • It’s job season: J-PAL, IPA and our friends post on a single portal here.
    • J-PAL’s Claire Walsh talks to the 80,000 Hours podcast (focusing on meaningful careers) about her job working to help governments use evidence in making policy.
    • One of the lowest-profile but highest-impact jobs in all of development is working with Caitlin Tulloch and colleagues at the IRC calculating cost effectiveness of different programs around the world. (Here’s a podcast of her explaining what her group does and why it’s crucial).
  • I swear I’m not making this up. Need a way to confidentially track and re-identify people in your data? One group of sexual health researchers used participants’ “porn star names” (name of first pet and street they grew up on):

    Porn star names were unique to 99% of their 1281 respondents to the baseline survey, and adding month/year of birth was enough to provide 100% uniqueness. When re-contacted later, they were able to match 76% of respondents between the two surveys using only the porn star name, and using month/year of birth they could further match 96% of those who provided a partially-consistent porn star name. (h/t Lee Crawfurd, who else)

  • 3-month visiting program in Germany for post-doc researchers from sub-Saharan Africa. (via Macartan Humphreys)
  • Harvard Ph.D. candidate Heather Sarsons previously found female academics suffered a “co-authorship penalty,” with co-authored papers helping their careers less, compared to their male counterparts. Now Sarsons is making a big splash with her job market paper on surgeons. Practices of medical specialists are dependent on other doctors, often primary care physicians, referring their patients to them. Sarsons finds that when female surgeons have a negative outcome (a patient dying), they’re punished, in the form of fewer referrals, more than men. The referring physicians are also less likely to refer their patients to other women in the same specialty.

    I find that men would have to receive patients who are 70 percentage points riskier on unobservables for risk to explain the gender difference in a PCP’s reaction.

  • The much-loved Princeton economist Uwe Reinhardt died this week. He was acclaimed for his ability to cut through complexity when it came to understanding and explaining the healthcare system with humor and for his general menchiness. Those qualities come out in his other writing as well. For meaningfulness, read his thoughts from 2003 on what it was like as a child growing up in WWII Germany, and for humor, his lecture on understanding Korean TV dramas.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

Screen Shot 2017-11-10 at 12.01.15 PM

South Africans, trying to come to grips with the astonishing scale of the crisis, have adopted a once-obscure political science term, “state capture,” as a staple of even casual conversation … Yet previous examples of state capture have almost always involved a broad cast of protagonists: an entire industry, for example, or wealthy businessmen as a group. In South Africa, it may have been pulled off by a single family.


  • Data Colada offers some advice on how to preregister a study effectively.
  • Larry Summers offered some thoughts on the future of development at the Center for Global Development video here.
  • J-PAL is offering scholarships to students from sub-Saharan Africa for their new MIT masters in Data, Economics, and Development Policy. A prerequisite is completing their online 5-class MicroMasters degree.
    • Also in the world of online resources, Daniel Björkegren’s course materials for his Brown University “Big Data” approaches to human behavior are here.
  • Jess Hoel asked for and got a lot of responses of examples of papers for teaching how one program does or doesn’t generalize to another context.
  • Rachel Glennerster offers four different models of how evidence-based interventions have been expanded and saved lives, and 3 tips for funders on how to support it (my paraphrasing here:)
    1. Stay flexible, you never know where innovation will come from.
    2. Evidence is a global public good, support cracking the theory behind why something works, so the same underlying principle can be put to use elsewhere.
    3. There’s a lot of evidence out there, some of it subtle. Support having the people who know it work closely with government to help them take advantage of it.
      1. I’d add a sub-bullet, which IPA & J-PAL are trying to do more and more: Not everything works off the shelf (see generalizability thread above). Support those experts working with government to incorporate testing and adjusting into their policy-making process.
  • For organizations who want to try randomization, Berkeley’s Josh Kalla created a simple randomizer tool for spreadsheets. Just upload your spreadsheet and adjust the proportions you want in each of up to 4 groups. (GitHub here.)


And if you were wondering if behavioral or traditional economists differ when it comes to free-riding, according to the latest Freakonomics ep both seem cool with it as long as you call it something else:

LEVITT: Yeah, my friend and colleague Richard Thaler won the Nobel Prize in economics, which has been another joyous occasion on the University of Chicago campus and especially nice for me because Thaler and I have been close friends for a long time. We play golf together quite a bit, and even had some little spillover for me. Golf Digest — which I’ve always dreamed of, somehow being written up in Golf Digest — they decided they’d write up Thaler because he won the Nobel Prize and because he’s an avid golfer. I was able to tag along and be the third wheel. Maybe I’ll get a brief mention or some kind of a scrap in Golf Digest so I can cross that off my bucket list.

DUBNER: So the moral of the story is you’ve wasted thousands of hours on instructional golf? You just need to get your Nobel and then you’ll get your spread in Golf Digest?

LEVITT: Exactly.

IPA’s weekly links


  • IPA has an opening for a Country Director for our Sierra Leone and Liberia offices (above photo comes from the former). A lot of interesting projects are happening there and our offices there have historically worked very well with the governments. I’ll let Rachel Glennerster describe it:

Rachel Gennerster on twitter

But the best reason is the amazing staff, here’s Jishnu Das talking about the Liberia office’s recent high profile RCT of public-private partnership schools there:

Finding children who have left a school is like finding a needle in a haystack. In a country where only 42 percent have access to a cell phone, it’s heroism.

  • On Vox Dev, Thomas Fujiwara talks about how Brazil’s move to digital voting machines ended up changing the face of local governments and policy. Millions of non-literate people hadn’t been able to vote effectively on write-in ballots, but the interface of the new machines was friendlier to them (using pictures and numbers in addition to words). The influx of poor voters resulted in electing local politicians more responsive to them, a 34% increase in public health spending over four years, and more prenatal health visits for less educated mothers.
    • Contrast that with this review (PDF, p.15) of how the U.S.’s hasty move to electronic voting after the Bush-Gore “butterfly ballot” backfired. In 2002, Congress passed the “Help America Vote Act” allocating billions to help local districts by electronic voting machines. The market responded quickly with machines that ended up having poorer interfaces than traditional voting systems. Lab studies comparing the two showed the new machines produced about twice the rate of voting errors (1-2% on paper vs. 3-4% electronically), and probably made voting problems worse than before.
  • The Journal of Economic Perspectives has a section of 3 (open-access) articles on turning experiments into policy.
  • Three Dartmouth psychology professors have been suspended during an investigation into serious sexual misconduct. State police and the district attorney are also investigating, but it appears that they got involved not because the school reported it, but because the DA happened to read about the school’s internal investigation in the student newspaper.
  • Norway punches above its weight in international development, in part because of its commitment to spending more than 1% of its gross national income on aid (though in recent years some of that has been redirected inward towards refugee resettlement). Dan Banik & Nikolai Hegertun compared Norway’s vs. China’s approach to development spending in Malawi and Zambia. While Norway spends more on civil society and accountability, China tends to avoid direct politics and focus on infrastructure and agriculture.
    • Meanwhile, AidData has had 100 people spending five years trying to figure out exactly what China spends on international development (now more annually than the U.S.). The team finds in a new working paper that every additional Chinese project is associated with 0.7% economic growth. They’ve made the data public.
    • The same group put out a report last year using satellite measures of nighttime light to estimate that Chinese (but not World Bank) projects increase regional GDP, but that the Chinese-funded projects concentrate three times as much in African leaders’ home districts while they’re in power.

And, Japan’s Phillips Curve Looks Like Japan:

Screen Shot 2017-11-02 at 4.01.41 PMScreen Shot 2017-11-02 at 4.01.57 PM

(h/t C. Trombley)

Why you or your students should apply to the PhD program at UChicago Harris


Professors who advise students on PhDs, and students thinking about a PhD: if you read this blog, there’s a good chance you should consider the PhD in public policy at UChicago Harris Public Policy.

The Harris PhD has a cohort of about 10 students per year. It has traditionally been strong in several areas: applied microeconomics, political economy of democracies, energy/environment, health, and education. The core PhD curriculum combines traditional economics training (microeconomics, applied econometrics) with political economy methods (i.e. formal theory).

Recently we’ve grown the political economy of development and conflict group. This is partly thanks to The Pearson Institute here at Harris. Between the new concentration of faculty, and the emphasis on the most rigorous training in economics, political science, and field methods, we think Harris has become one of the best places for a PhD student to come if they are interested in the political economy of development, conflict, or crime.

For instance, my fellow political economy of development faculty at Harris include Maria Bautista, Ethan Bueno de Mesquita, Oeindrila Dube, James Robinson, Luis Martínez, Austin Wright, and Jeannie Annan. We’re currently hiring more faculty in this area.

Other scholars focused on international political economy or development at Harris include Fiona Burlig, Anjali Adukia, Amir Jina, and Konstantin Sonin. Also, my colleague Jens Ludwig founded and runs the UChicago Crime Lab. Plus we have a huge number of applied microeconomists and formal political theorists.

Meanwhile, there is lots of advising and courses across campus. Most Harris PhD students have committee members from economics, Booth, law or political science, and take courses in these departments.. For instance, in political science there are Ben Lessing, Paul Staniland and Paul Poast. In development economics more broadly there is Marianne Bertrand, Leonardo Bursztyn, Adam Chilton, Michael Greenstone, Chang Tai Hsieh, Anup Malani, Rebecca Dizon-Ross, and Alessandra Voena.

Besides the famed Chicago interdepartmental seminars, like the Becker Friedman workshop, we have weekly external speaker series in public policy and economics, political economy, and development economics. Our new development PhD course sequence will be taught by me, Jim Robinson, Leonardo Bursztyn, and Alessandra Voena, plus others.

There are lots of opportunities for funding field work for your dissertation. The Pearson Institute provides generous funding for graduate research into conflict. Jens Ludwig, Oeindrila Dube and I lead JPAL’s Crime, Violence and Conflict sector, and I coordinate IPA’s Peace & Recovery program. Students of the many JPAL and IPA faculty on campus are eligible for funds from a range of JPAL and IPA initiatives. Finally, the new Tata Center for Development is a also great source of funding for graduate student research in India.

An important note on preparation and entry requirements: Because of the economics and formal theory core, applicants need to have a minimum of multivariate calculus and statistics training to enter the program. Traditionally, linear algebra is strongly recommended and real analysis is an advantage.

We understand, however, that not all political economy of development and conflict applicants will have this training, especially those with more of a political science background. We strongly encourage you to apply, even if you are uncertain whether they have all the requirements. If we accept you, it will be because we expect you will be able to handle the rigorous first year core, or because we believe we can help you get the additional math preparation you need before or during the program.

As one of the co-Directors of the PhD program, I hope to see some of you applications come in this fall. If you feel that the application fee is a barrier to you applying, please contact the office of admissions to request a waiver. We do not want this to be a barrier to you.

But given the overwhelming number of applicants to these programs, it’s uncommon to speak with faculty in advance. Like most economics and political science PhDs, we do not expect you to have developed a relationship with a faculty member in advance. Merely explaining in your application letter which faculty you would like to work with and why is sufficient. The office of admissions can explain other questions.

Like a lot of faculty, I cannot keep up with all the individual emails and inquiries I receive. I’m sorry if you do not get a reply. My approach is to focus my energies on admitted students and my advisees. Because I can’t answer individual emails, I’ve written a huge number of PhD advice posts (see right). The most relevant for admissions include these:


IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

Screen Shot 2017-10-27 at 12.55.39 PM

The links are back from vacation. We may have a few back links to catch up on over the next weeks, so here we go:

  • Rachel Meager has public speaking tips for economists.
  • If you want to catch up on a Twitter conversation including me, Chris, and a bunch of other people responding to the Cuddy article on what replication fights in psych mean for econ there’s a 168-slide storify here.
    • I wondered if econ is happily driving along at 65 mph waving at psych as it heads towards its own cliff, because I think every field is unaware of its own blind spots. For econ, I thought it wouldn’t be small samples, but lack of attention to the survey questions and what their measurement tools are getting at (in my experience other social science fields sweat these details far more than economists).
  • But a few days later I was proven wrong. Stanford statistician John Ioannidis (known for the paper showing most medical studies are probably wrong), and colleagues came out with a paper showing most empirical econ studies are underpowered (using 159 meta-analysis data sets of 6,900 studies). In plain language that means they in fact didn’t have big enough samples to support the effects they claim:

nearly 80% of the reported effects in these empirical economics literatures are exaggerated; typically, by a factor of two and with one-third inflated by a factor of four or more.

The paper‘s part of a section on reproducibility in econ (all papers ungated), but there are some slides here in more accessible language. (h/t Lee Crawfurd)

  • If misery loves company, a Nature survey of 1,500+ physical scientists found that more than 70% had tried and failed to reproduce another researcher’s experiments, and more than 50% had failed to reproduce their own experiments. Of the group that had tried to publish replications, successful replications were more frequently published than unsuccessful ones.
  • But some good news, there’s a new journal devoted to just publishing replications in empirical economics.
  • Some non-academic jobs:
  • For the academic-types, graduate students can once again blog their job market paper on the Development Impact Blog.
  • But a good thread for current and future graduate students – a reminder not to pin your sense of self-worth on your academic career. Too much is out of your hands. (h/t Raul Pacheco-Vega)

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

Queen of Katwe Movie Still

Madina Nalwanga as chess champion Phiona Mutesi in the film Queen of Katwe.
Photo: Edward Echwalu/Disney

  • Recognizing that an increasing amount of development policy is being done in developing countries, the prominent British NGO Oxfam is moving its headquarters from the UK to Nairobi.
  • There’s some evidence that being exposed to relatable role models can improve performance in school or at work. A newly-published RCT compared the exam scores of secondary school students in Uganda who viewed Queen of Katwe, a movie about a girl from a low-income community in Uganda who becomes a chess champion, to those who viewed a placebo movie. The students who watched Queen of Katwe were more likely to pass their national exams than those who did not, and the effects were especially large for girls and students who had performed poorly on the exams in the past. The working paper is available here.
  • The Kenyan Supreme Court delivered its full ruling nullifying last month’s election, citing concerns over the accuracy of the results.
  • There’s a free online course starting next week on the science of early childhood development from an international development perspective, including what programs are effective for early childhood.
  • A report from a U.S. government agency found that over the past decade, the government made $63 billion more in tax revenue from refugees than the refugees cost the government. The report was mandated by the president in March, but the administration chose not to release it when the findings were revealed in July (The New York Times obtained a draft copy).
    • Over at the Center for Global Development, economist Michael Clemens reviews the research on the economic impacts of wealthy countries accepting refugees.

Links will be on a break for the next couple of weeks—we’ll see you back here in October.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • In The New Yorker, John Cassidy reviews a new free online open-source economics textbook, The Economy. From an international collaboration of economists, it focuses on newer, post-financial crisis ways of thinking about and teaching economics.
  • Case Western economist Justin Gallagher documents the bizarre fight he went through to get one research group at the University of Texas to turn over the public state data set it was holding, including a Freedom of Information request compelling them to turn it over. That research group was the only repository of data (the state had wiped many of its records after turning it over), and uses it to produce a number of publications.
  • Some people say that you only learn about your own culture when you’ve been somewhere else. Former NPR East Africa correspondent Gregory Warner and his wife, novelist Sana Krasikov, talk to parenting/family podcast The Longest Shortest Time about readjusting to the U.S. with a 5-year old son who’d only known life in Nairobi. He discovered American kids had a concept of “personal space” that didn’t make sense to him, and also had difficulty with learning how the Kenyan style of sharing (where kids would treat property much more communally, walking out of a friend’s house with their toy), differs from the U.S. implicit concept of sharing (getting your share, and then voluntarily deciding to offer some to others). Web or “Bubble Boy” episode from Apple.
  • In a clever design, Blair, Littman, and Paluck had two versions of a film produced in Nigeria, one with and one without the actors reporting corruption. They then had staff of a survey firm pose as film promoters, distributing copies to local film sellers in randomly selected communities with instructions to give them away to customers. The film distribution was followed by a text message campaign, and resulted in boosted corruption reports, and changed attitudes toward corruption as measured in surveys.
  • It’s been reported in the past that the CIA’s ruse to find Bin Laden involving collecting DNA with a fake vaccination campaign led to a backlash against vaccinations in Pakistan. Now the measures are in, Martinez-Bravo and Stegmann find a one standard deviation increase in support for an Islamist party in a region is associated with a 9-13% decline in immunization rates after the vaccine ruse was reported.

And finally someone’s put all that inequality research to good use and calculated the gender inequality of likes on Tinder:

It was determined that the bottom 80% of men (in terms of attractiveness) are competing for the bottom 22% of women and the top 78% of women are competing for the top 20% of men. The Gini coefficient for the Tinder economy based on “like” percentages was calculated to be 0.58. This means that the Tinder economy has more inequality than 95.1% of all the world’s national economies. In addition, it was determined that a man of average attractiveness would be “liked” by approximately 0.87% (1 in 115) of women on Tinder.


(Image credit above)

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.

Screen Shot 2017-09-08 at 10.43.02 AM

  • Results of a long-awaited and somewhat controversial evaluation of a public-private partnership to manage schools in Liberia were released yesterday by IPA and The Center for Global Development:
    • With their education system in pretty bad shape, the Liberian government piloted the Partnership Schools for Liberia (PSL) program, which contracted out day-to-day management of some government schools to a mix of operators, non-profit and private, foreign and local. The government still owns the physical schools, the teachers are required to come from the same pool of government-employed teachers, and still have to teach the government-mandated curriculum, but within those parameters, the school operators have broad flexibility. Fortunately, the government invited researchers to randomize which schools were put under private operation, so it could be studied.
    • Preliminary results from the first year showed that students at PSL schools spent twice as much time learning, and students learned on average about 60 percent more than in the public schools, but it was very expensive. While the government budgeted about $100 per pupil for PSL schools (already twice what’s spent on public schools), the outside orgs brought their own funding, adding an extra $57 to $663 (or $1,052, depending on which numbers you use) per student.
    • The biggest operator spent the most and got the among the highest learning gains, but also removed more than half of their teachers and a number of their students (presumably, sending them back out into the traditional public schools).
    • The research team concludes that this may be a way to get learning up in an individual school, but probably not sustainable as a national program without significant changes.
    • In contrast, J-PAL reviewed evidence from other countries showing similar learning gains at much lower cost from a variety of other interventions. (With the caveat that they haven’t been tested in Liberia.)
    • Read more on the blogshort brief, or full working paper from Romero, Sandefur, and Sandholtz, or this thoughtful tweetstorm analysis from Abhijeet Singh.  It’s Mauricio Romero’s job market paper, so if you want to hear more, he’s probably happy to come talk about it.
  • J-PAL Executive Director and new Chief Economist for the UK aid agency DFID, Rachel Glennerster, with Claire Walsh, wrote a blog post about something that makes researchers break out in a cold sweat. What if you ask something people have always asked about using standard questions (in this case, womens’ empowerment), but also ask in a slightly different, more specific way, and find you get a very different answer?
  • Highlights via Damon Jones from this week’s twitter panel on balancing econ professional life with the rest of it. See the full discussion by searching twitter for the #EconLife hashtag.
  • Rebecca Thornton has put together a number of resources on gender and economics.
  • Cash is all the rage in the development and aid community. Parts of India are transitioning from subsidizing food for the poor to depositing cash directly into their bank accounts. But there have been some problems with the rollout, particularly with it not making it into people’s accounts, or it being difficult for recipients to access the accounts. Despite this, recipients say they still prefer the cash, which offers them flexibility in how much and what kinds of food they can buy.

Why I am not blogging anymore

Some of you have asked. I haven’t been blogging for many months, and I don’t see myself starting again in the next 6-9 months. But I am fairly sure I will start again in 2018.

“There are not enough hours in the day” was always a problem, and so it can’t explain the change. “I’m taking a break” is part of the story. But there are better explanations.

I am writing a book. Dani Rodrik stopped blogging for the same reason. He said that his brain only can think and write about so much at one time. I am pausing for the same reason. I woke up one day in July and unexpectedly wrote a page, and I didn’t stop. I am 100 pages in now, which amazes even me. The best one-line description is the subtitle of the course I’ve taught for 10 years: Why are some places poor, violent, and oppressive, and what (if anything) can you do about it?

Supply problems. I used to get amazing things to blog about from Google Reader. When that shut down Twitter was an ok substitute. But since early 2016 it has become a cesspool of people retweeting and liking things that burnish their self image and signal who they want the rest of the world to think they are. Most of that is about US politics, and it bores me. The newspapers have gotten worse. The New York Times and Washington Post have become crap, and even hopeful sites like Vox.com have begun to disappoint. They all respond to retweets and likes and not what is important or interesting in the world. I have yet to come up with a substitute way to consume the best new stories out there. ( I welcome suggestions.) I try unsuccessfully to filter out people who tweet US politics. Instead, I read books more, which is good. I recommend it. But those are cumbersome to blog, so it does not solve the blog problem.

Substitution. Ironically, I now tweet a fair amount. Tweeting is easier and often feels sufficient. Of course it is not, because tweets are fleeting and buried in an onslaught of Trump news, people favoriting the same old shit, and other silliness. I wish WordPress made it faster and easier to blog away from a computer.

Also, few people read blogs anymore. Including me. I no longer feel the pressure to write often, because the person who comes directly to the page daily or weekly in search of something new is a dwindling breed. Most people reach blogs by twitter and facebook. This has taken the pressure off of me and, what can I say, I respond to incentives.

If I should ever get the book done, I suspect the incentives to turn towards blogging again. In the meantime please keep enjoying Jeff Mosenkis’ amazing weekly links. Or wallow in the cesspool of crap with me on Twitter, @cblatts, more commonly known as “following”.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


A Red Cross pamphlet from WWI slogan (at the bottom): “Millions for Relief, but Not One Cent for Administration”

  • In a surprise ruling a few hours ago the Kenyan Supreme Court voided the outcome of the recent election, calling for a new one within 60 days. The Nairobi stock market dropped 10 percent right away, triggering a brief halt in trading. Follow Ken Opalo for the latest (and just in general).
  • Here’s one way to cut through IRB paperwork. Investor Peter Thiel had 17 patients flown to St. Kitts to inject them with an experimental herpes drug which couldn’t get funding or IRB approval in the United States. So next time the IRB does’t approve some of your survey questions, consider flying the village to the Caribbean to ask them there.
  • If you hadn’t heard about it yet, ProPublica and NPR some time ago did stories on the American Red Cross’ repeated failures and lack of financial transparency in disaster responses over the years. They included driving empty trucks around during Superstorm Sandy for news crews to film, and raising half a billion dollars for Haiti rebuilding, but only building 6 homes. You can get the history here, but I did not realize that 71% of their revenue is from for-profit blood services. That business has been squeezed in recent years by lowered demand, leading them to cut back on disaster staff.
  • Development Impact blog links are back from vacation!
  • Over at the Center for Global Development, an assessment of the state of health evaluations. In a blog postbrief, & full paper, Raifman, Lam, Keller, Radunsky & Savedoff describe finding 299 evaluations in the health sector and grading 37 of them in depth for quality. Results were disappointing.
  • The new Rough Translation podcast from NPR went to the DRC to look into what happened when NGOs started showing up looking to help survivors of highly publicized mass rapes. It created a cottage industry (they actually have a phrase that translates into that), of villages finding women to say they were raped to get the aid. But the show looks a little deeper with the journalist who originally investigated it into the morals of the issue. The NGOs aren’t going to question the veracity of a victim’s story, and both the NGOs and people there are afraid that if it comes out that the rape stories aren’t true, it will cut off the flow of aid to people who need it.
  • Kremer and Rao slides on behavioral economics in development are here if you missed them.

The economics to sociology phrasebook is fun (h/t Chris):


IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • Two podcast recommendations:
    • NPR has a new podcast, Rough Translation, from former East Africa correspondent Gregory Warner (web, Apple). It looks at how questions we deal with here play out differently in other cultures. The first episode looks at how Brazil ended up with race tribunals to evaluate who was Black enough to qualify for affirmative action. The second looks at fake news planted by Russia in Ukraine.
    • The fun “Tell Me Something I Don’t Know” podcast recently had a behavioral science all-star cast with Dean Karlan, George Loewenstein, Katy Milkman and a  gang of others (Web: Episode 24 “Behavior Change: Ultra Egghead Edition,” or Apple)
  • For economists (but probably useful for other academics as well), there will be a twitter panel discussion on balancing the professional part of your life with the rest on Wednesday, September 6th at 9PM EDT, dinnertime Pacific. You don’t have to be live to participate though; tweet your questions in advance with the #EconLife hashtag and the panel will answer.
  • A nice simulator complete with cartoons where you can play econ trust games. You can try different strategies to watch how outcomes change (and then set it to work simulating repeated play using different strategies). h/t Osman Siddiqi
  • How many social programs work? Depending on how you count, 60-80% of published evaluations don’t find a significant effect, but remember that published evaluations are not a representative sample of all social programs.
  • Google Earth is giving tours of indigenous and traditional homes around the world.
  • The Secret Economic Lives of Animals. Increasing numbers of species have been found to vary exchange rates for services (like wasps offering nesting space for childcare), based on changes in local supply and demand.
  • Two short and interesting econ lessons:
    • How the infamous Smoot-Hawley Tariff, in which the government ignored the warnings of over a thousand economists, sparked a global trade war which exacerbated the great depression. It all started with a move to protect the Mormon Church’s interest in sugar beets. Along the way they explain why trade protectionism is politically popular, even though it’s ultimately harmful to everybody.
    • How Pigouvian taxes work, in which a government taxes something that hurts others, allowing the parties involved to figure out how to minimize it, like a carbon tax.

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • You might have heard that just giving the poor cash, no strings attached, is all the rage in the effective aid community. Some people have suggested that if organizations want to give (more expensive) in-kind aid (food, cattle), they should first show that it’s more effective than cash. Dev Patel just recirculated a relevant paper (summary here) from Cunha, De Giorgi, and Jayachandran, who tested giving cash vs. in-kind food aid in very poor rural villages in Mexico. They found in this case the food helped the poor more for the outcomes they were looking at, general price of food in the village. While the cash increased prices a bit (not significant), trucking in food increased how much there was available in the village, lowering prices for everybody. This effect on prices was also stronger the more remote the village was.
    • Their findings underscore a point that even the most ardent cash supporters point out but often gets lost in the conversation. A prerequisite to cash working is functioning markets – cash doesn’t do you much good if there’s nothing around to buy with it, or it’s expensive because there’s not much of it. One of the attractive points of cash is its easy delivery to remote areas, but it’s worth remembering that remoteness may also hinder its effectiveness. (More on food and nutrition outcomes from that study here.)
  • At Slate Star Codex, Scott Alexander reports back on going to a conference for the Effective Altruism movement, full of people trying to figure out how to do the most good with their lives. He’d discovered that his profession, doctor, ranked relatively low on the altruism calculations, but also found that when you start thinking about how one person can alleviate the most suffering possible, it can get overwhelming quickly. Some groups literally get into the weeds – extending concern over the suffering of animals to include billions of insects in the wild.  
    • In the end, he comes away with a Tyler Cowen-style takeaway: it’s probably unrealistic to aim for a perfectly effective/altruistic world, but it’s not too much to hope for people to examine their lives and do a little better.
  • Alexander also had a very nice back and forth post Google memo with Wharton psychologist & NYTimes contributor Adam Grant. They dive into what the meta-analyses actually say about gender differences in skills and career preferences and have a back-and forth in the comments. It’s a long but interesting read, and nice to see a constructive and respectful argument about the data on the internet.
  • J-PAL has an impressive new bulletin combining effectiveness and cost of 58 different education interventions, and ranks how much education per dollar each provided.
    • In an accompanying blog post Rachel Glennerster points out most systematic reviews don’t break down results by gender, but they found in most cases girls benefitted more than boys.
  • But the best education program might be to just live in Canada. If Canadian provinces entered international education tests separately, three of them would be ranked in the top five countries on science test scores
  • I can’t believe I only recently found out that Kim Yi Dionne has a podcast, Ufahamu Africa, about politics and events on the continent.

And not only are Canada’s schools better than ours, even their roads are behaviorally behaviourally designed:


IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • Jobs:
    • NPR’s Planet Money is looking for someone who knows about econ to do shorter stories linked to the news of the day (explaining the econ of current issues in the news). Good communication/explainer skills but no previous journalism experience required. I know several of the people there and they’re all amazing, I can’t recommend them enough.
      • (I would just caution, from experience, that journalism culture differs from academia. It moves fast and requires precision under hard deadlines so you often have to become an expert in a topic by 4PM, and be exactly right, which doesn’t leave a lot of room for unpacking nuance.)
    • Activision is looking for an economist to run experiments on Call of Duty’s in-game economies. (via Jess Hoel)
    • The UK’s The National Institute of Economic and Social Research is looking for two economists to work on ed and labor policy studies.
  • Susan Dynarski started two great threads online on what books inspired people to get a Ph.D. in their chosen academic fields, and advice for first-year Ph.D. students (here’s a collection of the latter).
    • It’s worth noting a fair amount of advice involves taking care of your own mental health; one university survey found a third of Ph.D. students at risk for common mental health issues like depression.
    • See also this from Dina Pomeranz about all the students quietly thinking they’re the only ones with anxiety/depression, etc. and the subsequent discussion on coping strategies.
  • Seema Jayachandran did a very popular Reddit Ask Me Anything (more readable version here) about her Science paper on cash transfers for not cutting down trees (the AMA landed her on the front page of Reddit for the second time). In answering questions from the public, she was struck by how many people people had a moral objection to paying people not to do something (as opposed to traditional conditional cash transfers which reward people for doing something, like enrolling their children in school).
    • Similarly, NPR reports that despite an evaluation showing massive benefits to giving poor people cash in Zambia, moral objections from the public to giving “lazy” people free money limited the program eventually to just the “deserving” poor, such as the elderly, and people who can’t work.
    • Rich countries aren’t immune to this kind of thinking. A Vox The Weeds podcast (and parallel article) on the legacy of welfare reform from last year talks about how U.S. social safety net policy changed based on the public’s image of a single mother. At first, the U.S. image of a single mother was a widow trying to raise her children by herself. At that time it was seen as virtuous to help her stay home and raise her kids. When the public image of a single mother changed to a poor minority woman, programs began to see her as someone who should be out working and the design of the benefits changed.
    • Those of us who work in the world of evaluating the economics of anti-poverty programs are used to thinking about effectiveness and cost as the primary determinants policymakers need to know, but these are good reminders that the moral view of the design of the program may be just as important in determining whether a program gets implemented or gathers dust on a shelf.
  • The U.N. World Food Programme has had success with automated SMS surveys and is developing a chatbot to ask people in remote areas more detail about their food situation (prevailing prices and the like). They hope that AI will be able to parse and aggregate the information to allow human operators to disperse aid faster and more effectively.
  • This Week In Africa has all the roundups of the Kenyan elections (and Rwanda, and South Africa) you could want. But if it’s not enough you could check in with Ken Opalo, who says Kenya Deputy President William Ruto and the Jubilee party are well positioned for the future.

I mentioned up top the importance in journalism culture of speed and accuracy. USA Today used a photo of the wrong Asian in a story about implicit bias.


(Image credit above)

IPA’s weekly links

Guest post by Jeff Mosenkis of Innovations for Poverty Action.


  • This week’s Freakonomics episode, titled “Everything You Always Wanted to Know About Money (But Were Afraid to Ask)” (Apple podcasts) features an all-star cast of Jack Bogle on not trying to beat the market, Annamaria Lusardi on teaching basic financial tips to NFL players, and Harold Pollack on his index card of financial heuristics. Readers of this blog are all financial whizzes but, you know, for your friends and stuff.
  • Pollack explains at the end that he was perceived as political by some for his final rule – support the social safety net – and explains why his middle class family needed it. The social safety net is complicated but explained well in the Marketplace podcast series The Uncertain Hour (just 7 episodes, Apple podcasts). In one episode, for example, they take an honest dive into the debate over whether welfare should have work vs. higher education requirements (the former won out), and what the limited data we have says on the effectiveness of each.
  • VoxDev is going like gangbusters. In advance of the Kenya elections, Tavneet Suri talks about her study from the 2013 elections, and why her get-out-the vote efforts worked, but may have had some unintended consequences.
  • This coming Monday at noon eastern, Seema Jayachandran is doing a Reddit Science Ask Me Anything on her RCT using cash to slow deforestation in Uganda.
  • From the Center for Global Development, optimism on the new USAID administrator nominee. But USAID’s great innovation hub, Development Innovation Ventures, has suspended accepting new applications, with the agency citing other funding priorities.
  • A deep dive into why much of Haiti still doesn’t have a sewer system, despite outside efforts to build them. Among the critiques: reliance on outside aid makes local officials more responsive to what the donors want. But interestingly, the U.N. bringing cholera to Haiti might have actually spurred local agreement. according to the local sanitation director:

    “Only the cholera could make us have [the first sewage treatment facility],” Petit says. “Only cholera. Because we were afraid, totally afraid of cholera. For this reason, everyone agreed.”

  • The story of how Chile tried to use early computer networks and teletype to continuously monitor its economy in the early 1970’s, and how the CIA tried to stop it. (In the podcast, or on Apple, scroll all the way back to the first episode “Nineteen seventy three”)


[Image source above]