Replication data (and the perils thereof)

IPA and J-PAL have a new data replication archive. Most or all of my published papers make available full, original survey datasets in addition to the paper replication data, and I’m starting to archive all the data in the new IPA/JPAL dataverse. Here is one. Highly recommended.

I do wonder a little about perverse incentives for people replicating the paper, however. Here is a reply from Stefan Dercon and coauthors to a 3ie-sponsored replication exercise.

In this reply, we explain why we welcome the principle of replication studies. We document how we have approached cooperation with the replicator but also express disappointment in how this process seems to have operated in practice; specifically the extent to which it created incentives to go beyond replication until methods and data were found that yielded different results.

We are glad to note that in terms of pure replication, our results are confirmed beyond a minor coding error that did not matter for either the results or their interpretation. We are disappointed, however, that the replication study is selective in reporting our own cautious discussion on method and robustness in both our original AJAE paper, and a subsequent paper in Journal of Development Studies. We quote our own papers on how we already addressed a number of the concerns raised in this study and why we judge these innovations as being difficult to consider as ‘superior’ both in principle and in the way they are applied.

The study places considerable weight on the robustness of our results on agricultural extension but ignores that we have highlighted as much in both papers before. We are not convinced that there is much value added in the part of the study that investigates robustness rather than just replicability.

Berk Ozler has commented previously, and I thank the Development Impact blog for the links.

Update: Great thoughts on replication etiquette by 3ie’s Annette Brown

Is your college hoodwinking you?

Currently, the federal government gives just 24 cents in postsecondary education improvement grants for every $100 in grants for research.

From an NYT op-ed yesterday on the many deficiencies of college education.

Various solutions are proposed: more funding for better learning outcomes, more data for students and parents on college-specific outcomes (like graduation rates), and making some colleges responsible for the high levels of student debt and default.

Some thoughts:

  1. It’s actually pretty shameful that so few colleges incentivize or enable good teaching, especially the best ones. Incentives are not a terrible idea.
  2. That said, I feel like the op-ed posed solutions without diagnosing the problem. Why is it, exactly, that so many students (and parents) don’t make the single biggest purchase of their life without knowing or demanding more from their colleges? Why isn’t that the financial incentive colleges need?
  3. For a lot of people, their implicit answer seems to be: “Because potential students have poor information, they are enabled by cheap debt, and so for-profit colleges hoodwink people on a mass scale.”
  4. I don’t find this a very convincing diagnosis. But I do not know what the sensible alternatives are.


Links I liked

  1. Freakonomics interview with World Bank President Jim Kim
  2. Limited airborne transfer of Ebola is “likely”
  3. Does FT win headline of the year? “Political ambitions limited orgy involvement”
  4. My favorite hashtag at the moment: #HumanitarianStarWars
  5. “The Poetry of Richard Milhous Nixon”, composed of direct quotes from the Watergate tapes
  6. How a scarcity of security affects your brain
  7. What Gary Shteyngart learned from watching a week of Russian TV

What I’ve been reading

  1. Euphoria, by Lily King. Pioneering anthropologists in the field, making it up as they go along: The novel. Inspired by Margaret Mead. Probably the best book I’ve read this year. I find it impossible to imagine the equivalent book on economists or political scientists. 
  2. The Whites, by Richard Price. Lush Life is one of my favorite crime novels, and this new book doesn’t disappoint. He’s famous for his portrayal of cops. Extra alluring if you live in New York City.
  3. My Struggle, Book One, by Karl Ove Knausgaard. I stand by my claim that Book 2 is the best book I read last year. A memoir of a selfish writer. It stands alone. Book One is good but less insightful, less philosophical, less startling, less compelling.
  4. Get in Trouble, by Kelly Link. A collection of short stories that are an unorthodox blend of science fiction, fantasy, and horror by arguably one of the best writers of the craft.
  5. The Girl With All the Gifts, by M.R. Carey. Easily the most interesting and unconventional zombie book you will read. In my case, the only zombie book I ever have or probably will read. Nonetheless, a very good novel.
  6. Einstein’s Dreams, by Alan Lightman. Possibly the only top physicist who also has a literature appointment (at MIT in this case). The novel explores the nature of time through Einstein’s dreams. I like his non-fiction work much better, such as this book I mentioned earlier this week.
  7. Redployment, by Phil Klay. The wars in Iraq and Afghanistan told through the eyes of various military grunts. It’s been celebrated for its insights on the military experience. I can believe it. But the one chapter on the aid bureaucrat seemed over the top in its satire, so I worry that the military stuff is too. Still, a very enjoyable book.
  8. The New York Nobody Knows, by William Helmreich. An NYC professor walks every street in New York and writes about it. I had trouble getting into this book but love the idea. Maybe if I stuck with it I would have been rewarded.

I wondered when this would start to happen

Every seminar these days has pictures of program activities. So, of course, I take more photos than I used to in the past. This time in Uganda, the tables turned.

We were visiting a rather spectacular farmer. I whipped out my phone and captured his raised beds and different varietals for posterity. Then we walked off to look at one of his other fields.

At some point during the walk a nice gentleman with a pretty serious camera and camera bag joined our little group. At the last field, the farmer asked us to all pose with him for a picture. When I asked him why, he pointed out that people come, visit his farm and take his picture but don’t send him a copy. He wanted one of us, for his scrapbook.

Markus Goldstein’s latest notes from the field.

I am adding this to my personal cache of Real World Development Indicators, alongside “percentage of subjects that refuse to answer your surveys because they are too busy” and “number of government officials who give white researchers a ‘who the hell are you?’ look and listen to their national economist colleagues for advice”.

Ethiopia, by the way, is doing spectacularly on all counts, and Uganda is catching up. Liberia worsens by the day.

Other RWDI candidates?

In case you would like to feel very small today

NASA Releases a five-year time lapse video of the Sun that uses 2,600 Terabytes of data:

Which brings to mind the videos that zoom out and in from Earth to the known universe, where each circle represents a scale factor of ten (I like watching from 2:33 onwards):

Recently I was reading MIT professor Alan Lightman’s Accidental Universe, a lay introduction to modern developments in physics and some of the emotional, philosophical, and religious questions they raise.

In particular, I wasn’t familiar with some of the think around the multiverse–that this is just one of many existing universes where different laws of nature may apply.

If the multiverse idea is correct, then the historic mission of physics to explain all the properties of our universe in terms of fundamental principles—to explain why the properties of our universe must necessarily be what they are—is futile, a beautiful philosophical dream that simply isn’t true. Our universe is what it is simply because we are here. The situation can be likened to that of a group of intelligent fish who one day begin wondering why their world is completely filled with water.

That uncertainty also disturbs many physicists who are adjusting to the idea of the multiverse. Not only must we accept that basic properties of our universe are accidental and uncalculable. In addition, we must believe in the existence of many other universes. But we have no conceivable way of observing these other universes and cannot prove their existence. Thus, to explain what we see in the world and in our mental deductions, we must believe in what we cannot prove. Sound familiar?

So even the zoom out and in above is a fraction of what exists.

Some people find more comfort in philosophy and religion as a result, but I find much, much less.

Talk at MIT Feb 19th: Reducing adult poverty, crime, and violence through behavioral therapy

I’ll be giving the comparative politics seminar 1-230pm. Here’s the paper title and abstract

Reducing adult poverty, crime and violence through late-stage noncognitive investments: Experimental evidence from Liberia

by Christopher Blattman, Julian Jamison, and Margaret Sheridan

Abstract: What to do with poor, violent, criminal young men? We evaluate two interventions to reduce poverty and instability among high-risk Liberian men. We show that self-control and self-image are malleable in adulthood and that building such noncognitive skills reduces crime and violence. The main intervention was an 8-week program of behavioral therapy designed to reduce impulsiveness, manage anger, and increase self-discipline, by fostering skills and a noncriminal self-image. We assigned men to receive therapy, therapy then $200, $200 alone, or neither. Cash alone led to short-lived income gains, dissipating within months. Therapy, however, improved self-control and attitudes to violent and criminal behavior. This drove large, sustained falls in crime and violence, but did not affect other economic decisions (such as investment). The effects of therapy on crime and violence were greatest in concert with the cash, largely we argue because the short-lived boost to legal employment reinforced therapy’s behavioral changes.

What is your One Ring?

Jason Kottke relates a passage from an explainer video on the One Ring from the Lord of the Rings:

“First, the ring tempts everyone (well, almost everyone) with promises that yes, this little ring can be a mighty weapon or a tool to reshape the world and gosh don’t you just look like the best guy to use it. Let’s go vanquish the powerful demigod who lives over there to get started, shall we? This is why the hobbits made great ring bearers, because they’re pretty happy with the way things are and don’t aspire to greatness. Of course, there’s Gollum, who started out as a hobbit, but all things considered, he held out pretty well for a couple hundred years. Set the ring on the desk of most men and they wouldn’t be able to finish their coffee before heading to Mordor to rule the world and do it right this time.”

I enjoyed Kottke’s take:

What’s interesting about hearing of The Ring in this focused way is how it becomes a part of Tolkien’s criticism of technology. The Ring does what every mighty bit of tech can do to its owner/user: makes them feel powerful and righteous. Look what we can do with this thing! So much! So much good! We are good therefore whatever we do with this will be good!

The full post is interesting.

If there were a way to get every Kottke post by email (is there?) this is probably one of the five blogs I would subscribe to, so as not to miss a post.

Are these the best Amazon reviews ever? (Anti-vaccination edition)

61bNGvoLVYL._SX258_BO1,204,203,200_Salon reports:

In 2012, a proactive Australian anti-vaxxer named Stephanie Messenger self-published a children’s book called “Melanie’s Marvelous Measles.” With the book, Messenger endeavored to “educate children on the benefits of having measles and how you can heal from them naturally and successfully.”

It also highlights some truly, truly wonderful Amazon reviews:

“This book has been a wonderful distraction while I sit in the hospital to support my friend whose baby has this delightful disease. Since the child now has both pneumonia and encephalitis, I’ll have to check out the additional titles mentioned in Michael J. Gulgoski’s wonderful review. We’re going to be here a while. Unfortunately, I had to give this only one star because I hate the name Melanie.” –This Daydreamer

“Finally! A children’s book with an agenda I can get behind! I always thought I loved kids until I actually had one of my own and boy was I wrong! I researched anything and everything I could possibly do to get rid of the little brat, but I didn’t want to be arrested for murder and childhood cancer is just too darn unpredictable. Fortunately, I stumbled upon ‘Melanie’s Marvelous Measles’, and learned that there is a huge community of people who hate children as much as me! Thanks to Melanie, I was able to ignore my pediatrician’s recommendations to vaccinate my daughter before our trip to Disney World, all while acting like I want what is ‘best’ for my child.” –brittany

Amazon page is here in case you want to add your voice.

Brian Williams for The Daily Show? And Stewart for NYC Senate.

Unlikely, but consider:

  1. He did lobby to get The Tonight Show after Leno,
  2. He’s actually quite funny, and
  3. We now know he’s good at fake news.

The best imagined phone conversation, via the Reddit thread on Stewart.

NBC Mgmt: Hey Brian, what’s up?
BriWi: Jon Stewart just let me know on the DL that he’s leaving the Daily Show at the end of the year.
NBC Mgmt: Good. With Colbert in at CBS that’ll help with Jimmy and Seth’s ratings. How does that affect you?
BriWi: I want his job.
NBC Mgmt: What are you high? You’re the closest thing this generation has to a Walter Cronkite or a Dan Rather. Plus you’re under contract for the next two years.
BriWi: So you’re saying there’s no way you’ll let me out of my contract?
NBC Mgmt: You’d have to something so boneheadedly stupid it’d seriously damage your credibility and the network’s.
BriWi: Hmmm… I wonder…
NB Mgmt: What’s that?
BriWi: Hmm? Oh. Nothing. Goodnight, sir.
NBC Mgmt: Good night, Brian.

In the meantime, I can’t quite imagine the 2016 election cycle without Stewart.

So… let me add an implausible New York ticket to your 2016 dreams: Clinton/Stewart.

Unlikely. But someone will need to fill Hillary’s Senate seat.

When repression backfires: Germans in the USA

Can forced assimilation policies integrate immigrant groups?

In the period 1917–1923, several US states barred foreign languages from their schools, often targeting German explicitly. Yet rather than facilitating the assimilation of immigrant children, that policy instigated a backlash. In particular, individuals who had two German parents and were affected by
these language laws were less likely to volunteer in WWII; they were also more likely to marry within their ethnic group and to choose decidedly German names for their offspring. These observed effects were greater in locations where the initial sense of German identity, as proxied by Lutheran church influence, was stronger. These findings are compatible with a model of cultural transmission of identity, in which parental investment overcompensates for the direct effects of assimilation policies.

A paper by job market candidate Vasiliki Fouka.

When did extreme poverty end in today’s “rich world”?

The answer is “not all that long ago”

The very interesting Poverty Analysis blog points me to this graph and argument by Martin Ravallion (which I had missed):

mravallion1aFrom Ravallion writing on the CGD blog:

today’s rich countries had poverty rates in the early and mid-19th century that are comparable to those found in even relatively poor developing countries today.

…In most cases, their poverty rates fell dramatically in the 19th century (Japan was a late starter but caught up in the 20th century). Yet today there is virtually no extreme poverty left in today’s rich world, when judged by the standards of poor countries today.

when progress against poverty is measured as a % point per year it slowed down a lot toward the end (as can be seen in Figure 1).

More surprisingly, when measured in proportionate terms, experiences differed greatly… Some countries (the US, the UK, Japan) saw steady progress in proportionate terms, while others saw more erratic changes in rates of progress at low poverty rates.

While we often assume that it will be a long hard slog to get the last few percentiles out of extreme poverty, some rich countries maintained steady progress to the end, and some even accelerated.

My favorite, probably embellished, way of making the same point (rich countries recently had horrendous poverty) comes from George Orwell’s Down and Out in Paris and London. A great, great book.

Other equally interesting data and bits at the Poverty Analysis blog.

Will my job soon be replaced by robots?

A new robot just discovered that a mixture of elements that can fight cancer can also treat malaria. Will artificial intelligence be able to unearth life-saving drugs quicker and more inexpensively than humans can?

Each day, the robot scientist checks whether 10,000 chemicals are toxic to humans. One human can typically screen only 10 to 20 chemicals a year.


Or maybe it will free us all to become literary critics and anthropologists.

Work for me as a Research Assistant in New York for up to 2 years

The job posting is here. It is through Innovations for Poverty Action (IPA) and J-PAL Please only apply or ask questions via the instructions on the IPA website, and not by emailing me, as I only see the short list and do not screen initial applicants.

The Research Associate will work closely with Chris Blattman at Columbia University on several projects related to poverty and violence.

Essential Duties and Responsibilities:

  • Manage data entry and assist in data cleaning
  • Supporting quality assurance measures for ongoing research projects
  • Conducting data analysis
  • Assisting in the writing of project reports, policy memos, and academic papers on a variety of topics
  • Corresponding with external partners organization and donors as needed

Education and/or Work Experience Requirements:

  • Bachelor’s degree in economics, mathematics, statistics, political science, or related fields

  • Strong quantitative skills

  • Excellent management and organizational skills

  • Knowledge of Stata (required); familiarity with R (preferred)

  • Experience as a research assistant

  • Extensive experience in applied statistics is a strong plus

  • Fluency in English with strong communication skills

  • Flexible, self-motivating, able to manage multiple tasks efficiently, and team player

  • 2 year commitment preferred

Social science is not a real science because it only published positive results?

Joachim Voth points me to a paper that looks for a “hierarchy of science” according to whether or not the discipline publishes null results.

Here’s how a sample of papers perform, by discipline:

Screen Shot 2015-02-04 at 9.46.12 AM

You can quibble with sampling, sample sizes, definitions, etc (and I would) but anyone who has had tremendous difficulty getting a null result published (I have, twice) knows that confirmation and publication bias is alive and well. If they simply got pushed to the good field journals, I could understand, but even there it can be tricky.

Even so, one reason to take the so-called hierarchy of science  with a grain of salt is the following figure:

Screen Shot 2015-02-04 at 9.46.30 AMEverything is driven by “pure” science, meaning (I think) the testing of theories and predictions from very basic science (think theoretical physics). The article is weak on definitions.

So the punchline is that empirical tests of highly theoretical models seldom pan out. Which is basically my experience in economics and political science too.

Even so, the next time you are asked to referee or report on a null result, give it a second chance.