Do open data badges work? In April of 2019, Psychological Science published its first issue in which all articles were awarded Open Data badges. Prompted by a tweet from open science activist Nick Brown, a group of twelve researchers set out to see if we could independently reproduce the published results from the materials provided. Ultimately, only one of the 14 articles could be exactly reproduced, with three more essentially reproducible with minor variations. I will talk about my experience being part of this project, and discuss recommendations for the future.