ASU faculty part of national summit tackling the lack of reproducibility in science research, a $28 billion problem in the U.S. alone

To err is human, even for one of the most number-crunching, rigorous and truth-seeking of all activities: science. The issue of errors in science — irreproducible research — was a focal point for a national discussion when the Global Biological Standards Institute (GBSI) hosted its 2016 BioPolicy Summit last week in Washington, D.C.

GBSI President Leonard Freedman and co-author Tim Simcoe, associate professor at Boston University, documented that up to 50 percent of published, pre-clinical research is irreproducible, with an estimated annual cost of $28 billion in the U.S. alone.

“These are big and troubling numbers,” said Freedman. “Over the long term, science is self-correcting. In the short term, the effects of irreproducible research include the cost of time and resources … and a growing mistrust in the biomedical research enterprise.”

There is also the untold human cost of delays in breakthrough discoveries, therapies or potentially new cures from the faulty data, reagents that don’t work as advertised or experiments that can’t be validated.

The 2016 GBSI Summit — “Research Reproducibility: Innovative Solutions to Drive Quality” welcomed premiere life science thought leaders, including Arizona State University biomarker researcher Joshua LaBaer and science correspondent and moderator Richard Harris (currently on leave from National Public Radio as a visiting scholar this spring at ASU), to explore the driving forces and profound impacts behind the issues.

“Despite tens of thousands of biomarker papers published, only about a half-dozen biomarkers have been FDA [Food and Drug Administration] approved,” he said. “That’s just a huge discrepancy in the number of papers published, and those that stood the test of time.”

Arizona State University biomarker researcher Joshua LaBaer (second from left) is partnering with the Global Biological Standards Institute to establish a pilot project creating a new gold standard and comprehensive registry of how each repository managed its sample collections.

To upend the scientific status quo and improve biomarker discovery and validation, LaBaer has led a massive effort to build fully sequence-verified clone sets for all human genes and other model organisms. The giant gene bank, called DNASU, is now managed in an automated repository with more than 250,000 samples, which are openly shared in more than 35 countries to benefit the worldwide scientific community.

Despite his pioneering efforts and countless other colleagues working hard to improve scientific reproducibility through cutting-edge, open-source solutions, the scale of the issue has required a system-wide re-examination of the current science climate.

“The problem of fixing reproducibility, $28 billion, is an amount that represents almost the entire annual NIH [National Institutes of Health] budget,” said keynote speaker Judith Kimble, Ph.D. at the University of Wisconsin–Madison and Howard Hughes Medical Institute investigator. Kimble emphasized, “The elephant in the room is hyper-competition.” 

Never before has the U.S. produced so many Ph.D.s, nearly 80,000 annually, to compete for so few federal dollars (a threefold increase in the number of Ph.D.s compared with a generation ago, with a twofold increase in the number of NIH grant applications). And this is even after leaders in Washington have just voted to give the NIH a huge increase in funds ($2 billion). But in terms of constant dollars, the amount is still down about 30 percent since before the recession, leaving the new crop of researchers to fend for less, noted Kimble.

With a flat funding environment, there is a tendency for scientists to propose what they know will work, instead of riskier research, the type that often leads to truly groundbreaking discoveries.

“Competition arises only when fields exist,” said Arturo Casadevall, professor at the Johns Hopkins Bloomberg School of Public Health. “Whenever there is competition, competitors try to use existing methods to solve the problem. Therefore, competition tends to put an end to innovation. It’s the opposite of what people often think.”

Unintentionally, research competition has led to a “more cooks, worse soup” scenario for science, as more researchers than ever apply for shrinking federal funds, creating a lack of transparency in sharing data, and low morale, ultimately giving researchers pause to understand how the key ingredients went astray.

But as Richard Harris poignantly pointed out during the panel discussion, since securing federal awards and producing top scientific publications are based entirely on a peer-review system, “We have met the enemy, and it is us!”

When the stakes are so high to publish in the best journals, with entire research careers at stake against more and more competition, it’s no wonder that each research lab tends to act like a top chef, hoarding their secret sauces and recipes.

“What’s good for me as a practicing scientist is not necessarily good for science,” said Brian Nosek, professor at the University of Virginia and executive director of the Center for Open Science. “The current system rewards scientists for publishing often and getting tenure. Open data and transparency are the keys to improving reproducibility.”

“Over the long term, science is self-correcting. In the short term, the effects of irreproducible research include the cost of time and resources … and a growing mistrust in the biomedical research enterprise.”
— Leonard Freedman, president of the Global Biological Standards Institute

“There is need for all of us to get our assumptions out there for people to see, but we need better carrots and sticks,” said Amy Herr, professor, Department of Bioengineering at the University of California, Berkeley. She has witnessed the stress of competition as it has even trickled down to affect the attitudes heard from the next generation of scientists. “When students say only that they want to publish in a prestige journal instead making a new discovery, my heart just breaks.”

To solve the reproducibility issue, the GBSI announced at the BioPolicy Summit the Reproducibility2020 Challenge, to ensure that major solutions are in place to improve reproducibility by the year 2020.

The action plan of the challenge calls for the entire research community to improve in three critical areas: reagent standards and validation, shared protocols and data, and improved training.

“Why many of us are working so hard in this endeavor is that we start with an understanding that data is not accurate,” said Herr. “That's just how we work.”

It’s also a key reason why LaBaer has now been partnering with GBSI to establish a pilot project, called BioSpecimen Commons, to create a new gold standard and comprehensive registry of how each repository managed its sample collections.

“Translational research depends on clinical specimens,” LaBaer said. “Unfortunately, researchers are rarely told how those specimens were handled before they are used, which creates variables that can dramatically affect the research outcome. Without that critical information, you never know if you are comparing apples to apples or to some other fruit. We cannot currently dictate how samples are handled, and that may not even be advisable, but we can ask that everyone describe exactly what was done. The first step in all of this is transparency.”

For the BioSpecimen Commons, every sample will be labeled with ID number(s) that link it to the specific recipes used in its preparation (standard operating procedures, or SOPs). In this way, the handling of every sample will be publicly available. LaBaer envisions as more and more people submit SOPs, then other people can look at them and decide which one is best for them. There will be a comments section, similar to Amazon, where people can comment and let the community know which ones worked and why. Over time, different SOPs will grow in popularity as researchers comment on which one worked best for them.

“Now, we are getting the whole community involved in saying how they did it. And every bit of data needs to go up on the web. If we make it transparent, at least we can take a step forward, and potentially amplify it in any area of biomedical research where there are protocols to follow.”

If more researchers can adopt such big-data, validated open science, it will not only help shape tomorrow’s discoveries, but bring more value to the health and well-being of society.

To learn more, go to GBSI’s Reproducibility2020

Top photo by Charlie Leight/ASU News

Joe Caspermeyer

Manager (natural sciences) , Media Relations & Strategic Communications

480-727-4858 joseph.caspermeyer@asu.edu

ASU researchers awarded grant to study why athletes fail to report concussions

When helmets clacked together during Sunday’s Super Bowl 50, the echoes might have been heard differently than in years past. Call it the concussion awareness effect.

In the 12 months since the last Super Bowl, the ongoing debate about football and brain injuries has only increased, with the conversation reaching different audiences thanks to the film “Concussion,” starring Will Smith as the doctor who discovered chronic traumatic encephalopathy (CTE) — which is believed to be caused by repeated head trauma — and how it was prominent among retired football players.

The discussion flourished again last week during the Super Bowl buildup when researchers at Boston University said legendary Oakland Raiders quarterback Ken Stabler, who died in July, suffered from CTE.

As long as there’s football, there will be concussions. And as long as there’s football, the debate about whether to allow children to play the sometimes-violent sport will remain. Now researchers at Arizona State University are trying to add a layer of understanding to the broadening discussion. Specifically, they want to know why some athletes, who are becoming better educated about concussion symptoms and side effects, don’t self-report concussions during sporting events.

The Arizona State University Center for Strategic Communication has been awarded a $400,000 grant from the NCAA Mind Matters Research Challenge to study how vested interests and team culture influence concussion reporting among college athletes.

“There are circumstances like wanting a pro career or letting the team down,” Corman said. “If they get their bell rung in a game, there might be a pro scout in the stands and they might lose exposure to that person. ... It’s a really complex situation. This is one of the first studies to examine this issue in this context.”

A “suck it up” culture, the desire to be a hero, and winners-never-quit messages contribute to what Corman describes as an attitude, behavior and communication issue.

According to a recent study by researchers at the Center for Organization Research and Design, even when faced with a 100 percent chance of brain damage by age 50 to play in the NFL, 15 percent of survey respondents — overwhelmingly former players, and many medically diagnosed with concussions — said they would take the risk.

The upcoming three-year study by Corman and colleagues will examine 500 Pac-12 athletes in football, wrestling, soccer, field hockey, lacrosse and basketball. Staff members will be interviewed about athlete narratives and team culture and values.

A targeted strategic messaging campaign to enhance athlete attitudes and behaviors about reporting concussions will be developed and tested in the third year. Strategies coaches and trainers can use to change team culture about head injuries will also be identified.

“We are excited about the project and think it will have a big impact on understanding and changing athlete attitudes toward concussion reporting, and enhancing team culture supporting those practices,” Corman said. “We can help make athletes safer and increase the long-term sustainability of popular contact sports.”

Scott Seckel

Reporter , ASU News

The dinner conversation at the Terrill-Pizarro household is often lively and robust, sometimes taking on the dynamic of good cop/bad cop. Sometimes literally.

William Terrill, who grew up in a working-class white suburban town in eastern Pennsylvania and was a member of the Military Police in the Air Force, saw the good side of police. Meanwhile, his wife Jesenia Pizarro, a self-proclaimed ‘Jersey girl’ who grew up in an inner city housing project, can share scenes of police brutality.

Together, they make an interesting couple.

The two ASU professors arrived from Michigan State University in January and study the flip side of criminal justice, often using their research to educate each other.

And sometimes not.

“When we first met in 2000 she told me that my dissertation on police restraint was essentially crap,” Terrill, a new professor in the School of Criminology and Criminal Justice, said with a laugh.

“I grew up where police were viewed mostly as positive and it took some time for us to see eye-to-eye. But often Jesenia demonstrates to me the other side based on her past experiences.”

That other side was an inner-city housing project in Newark, New Jersey, where Pizarro grew up. It was where she saw up close how crime, drugs and undue force — which was also displayed by police — ruled the neighborhood.

“I knew drug dealers. I knew people who got shot. I knew people who were homicide victims,” said Pizarro, an associate professor in the School of Criminology and Criminal Justice. “And interestingly enough, you always knew someone who were beat by the cops or allegedly mistreated by police.

“I wanted to do something and had this vision of getting the bad guys because I was surrounded by it all the time. As a kid I often wondered why people do what they do and what can we do to make it better?”

The two professors, who married in 2005, have made life better for their students, police departments and the public through their research.

"I don’t do research just to get my work published in a nice journal. I want my findings to have real meaning and do something for that community.” — Jesenia Pizarro, associate professor in the School of Criminology and Criminal Justice 

Terrill’s research on police restraint and use of force is complex, insightful and timely in light of the crisis of confidence about the integrity of law enforcement over a string of shootings across the U.S. — some of which were captured by citizen video and viewed nationwide.

“A lot of my research, even dating back to the mid-1990s, shows that three out of every four police encounters with the public shows that they were using less force, not more. Police were often de-escalating as a citizen was trying to escalate at the verbal level,” Terrill said. “They were using persuasion, their voice commands and the very thing we ask officers to do. Police are very good at their jobs and use great restraint but it still leaves us with a percentage who aren’t doing those things and stepping over the boundaries, the legal lines, the policy side.”

Terrill said police work is ugly by nature and when it gets caught on camera in urban distressed areas, it raises uncertainty nationwide.

“People start to think, ‘I can’t believe what I’m seeing. Maybe my perception has been wrong all this time,' ” Terrill said. “There will be short term growing pains and with that I think the long term gains will be much more positive than we’re seeing now. Body cameras and dashboard cams will provide more transparency and then there will be a level playing field for citizens and police officers. Police who may have a tendency to go over the line now may say, ‘I may get caught now, so I won’t do that.’ ”

Pizarro mainly studies homicide and why people kill. She says most of homicide research focuses on the socioeconomic issues, but she is looking in a different direction.

“I’m much more practical and focus on the proximal causes; what will increase the likelihood of being killed,” Pizarro said. “What are the immediate odds that will increase the likelihood of homicide? Lifestyle, location, situational events, demographics, owning a gun — issues that help me get at the real problem. I don’t do research just to get my work published in a nice journal. I want my findings to have real meaning and do something for that community.”

Homicides are down and so is violent crime as a whole, Pizarro said, adding that crime is like an accordion and comes and goes in waves.

“If you look at the history of mankind as we’ve become more civilized, we’ve become less violent. If you look strictly at homicide rates, this is the safest the country has been compared to 10-15 years ago,” Pizarro said. “Having said that, certain areas, mostly in the inner cities, have not felt that decrease.”

These days Terrill and Pizarro say they have also become more civilized when it comes to their points of view, each one coming around to the other’s way of thinking.

“As I have grown older and gained more experience, I don’t discount the fact that there are still bad cops out there,” Pizarro said. “But I’ve also had the opportunity to learn there are far more good police officers than bad. I have seen homicide detectives put in many extra hours trying to crack a case and officers patrolling communities who have good interaction with the people in the neighborhoods.”

For his part, Terrill says he has seen police departments resist change and not so keen on taking advice from outsiders. However, he says the last two years have forced departments to change, or else.

“There’s plenty of progressive police departments prior to this current problem but the ones who are holding onto the old ways — paying lip service to trust and community policing — are now saying they can no longer avoid it anymore,” Terrill said. “A great local example is Sheriff Joe (Arpaio, Maricopa County Sheriff). He was dead set against body cameras and then all of a sudden he made a comment recently, pretending he wasn’t against it at all and that his department has been at the forefront of the issue. Even he’s saying without saying it, ‘I can’t resist this movement of accountability.’ ”

One thing Terrill and Pizarro can agree, ASU is a place where they can make a difference.

“I love the urban feel of the Downtown Phoenix campus and it’s a great place to be a criminal justice professor,” Terrill said. “It brings a lot of research opportunities because you have many mid-to-large sized law enforcement agencies here and I know it will yield a lot of great work.”

Pizarro echoed the sentiment.

“The name of the college alone makes me want to go out into the community and translate research into real life,” Pizarro said. “I’ve always wanted to make a difference and let’s face it, inner city girls like me who grow up in the hood rarely get to a professor at a top research university. It just doesn’t happen.”

ASU professor says to build 'teaching and learning communities'

American Founding Father Benjamin Franklin expressed in characteristically concise and pithy fashion his view on effective education: “Tell me and I forget. Teach me and I remember. Involve me and I learn.”

The statement could serve as an abbreviated description of Arizona State University professor Stephen Krause’s goal in his work to develop and implement teaching and learning practices that engage and motivate teachers and students alike.

"It is a no-brainer,” Krause said, that engaged and motivated students and teachers are essential to successful education. The big challenge is to come up with ways to effectively instill new attitudes and approaches necessary to ensure engagement and motivation can be achieved and sustained.

Krause teaches materials science and engineering in ASU’s Ira A. Fulton Schools of Engineering, and for the past eight years has also been designing and refining an approach to education called Just-in-Time-Teaching with Interactive Frequent Formative Feedback.

That has meant providing a variety of meticulously detailed strategies and techniques aimed at realizing Ben Franklin’s educational ideal. It has also involved establishing evidence that results of using this method can be rigorously analyzed and objectively evaluated to gauge its success or expose any need to alter the blueprints.

The project has shown enough promise that the National Science Foundation recently awarded Krause a $1.5 million grant to expand the endeavor. In the next three years, he will work with more than 80 ASU engineering faculty members to put the Just-in-Time Teaching approach into action.

They are to be trained in how to establish productive “communication channels” between themselves and students, something that requires teachers to transition “from being the sage on the stage to being the guide on the side,” Krause explained. “It is about teaching teachers not just how to teach certain subject matter but how to teach students how to learn it.”

This requires knowing how to foster collaborative learning in the classroom, creating an environment in which students teach each other with guidance from the faculty member.

“The class becomes more than a transfer of technical knowledge from professor to students. It becomes a social activity that evolves into a learning community, and it’s this kind of cooperative environment that is motivating to students and teachers,” Krause said.

Seeing the relevance of classroom lessons

The social connectedness and collaborative spirit also extends to the relationships among teachers who will work to institute the change in approach.

For the project to be successful, “faculty support for each other is a big thing,” Krause said. “They have to talk to each other about issues and problems that come up, and work together on solutions.”

The overriding objective is not only to develop camaraderie that enhances teaching and learning. The foremost focus is on communicating to students the relevance of their classes.

“In all the equations, calculations, formulas, data, graphs, charts and the stuff in the textbooks, students must be made to see the possibilities of things they can get excited about doing,” Krause said. “They need to understand the social impact engineering and science can have and be able to envision their future being a part of it.”

The new style of teaching and learning emphasizes enabling students to frequently put to use the technical knowledge they’re being given in the classroom.

Teachers would encourage students to explore how they might be able to contribute valuable things to the world by using what they’re learning in class.

“Engaging in hands-on activities generates the kind of thinking you use in engineering design, and that is where motivation sets in and real learning happens,” Krause said.

Evaluating effectiveness of teaching methods

The project will include regular and stringent evaluations to determine if the new method is producing intended results.

How well faculty members adapt to the new approach will be assessed through classroom observations by graduate student researchers and through faculty surveys about changes in their motivation, attitude and social connectedness to other faculty members involved in the project.

In addition, an automated web-based system will give students the opportunity to provide feedback on their classes and how effectively they think teachers are boosting student learning.

Evaluations will also assess changes in students’ attitudes and motivation, and measure how the changes affect students’ persistence in completing courses.

These performance assessments will be used as a basis for formulating steps to improve course content, teaching techniques and materials.

Expecting significant progress

Krause said there is no getting around the fact that learning engineering and science is difficult, and that the demands of mastering the disciplines will always lead some students to abandon the pursuit. But he thinks the Just-in-Time-Teaching approach can help keep more students from dropping out of even the tougher courses.

His confidence is bolstered by studies of the outcomes of two classes at ASU and two at other universities that have employed Just-in-Time-Teaching strategies. Few students dropped out of the courses and their grades were notably higher than is typical for most classes. Grades for poor performance and failing grades dropped by half from what had been normal for those courses. Retention rates in the classes jumped from about 80 percent to 95 percent.

With the teaching and learning framework he has developed, Krause says, “I think we can hope to achieve similar success at ASU. I believe we can successfully scale it up, not just to involve more faculty in engineering but to transfer it into teaching in other disciplines.”

Project will be team effort

Helping him attempt to achieve that goal is a team of co-principal investigators for the project.

Fulton Schools faculty members on the team are James Middleton, a professor of mechanical and aerospace engineering, Keith Hjelmstad, a professor of civil, environmental and sustainable engineering, and biomedical engineering lecturer Casey Ankeny.

Others ASU faculty involved are Robert Culbertson, an associate professor of physics in the College of Liberal Arts and Science, along with associate professor Eugene Judson and assistant professor Ying-Chih Chen, who teach and do research on science education in the Mary Lou Fulton Teachers College.

“Having this team of colleagues to work with has been critical to developing the project,” Krause said. “The range and depth of their expertise in what it takes to truly educate students is what made it possible to put together a comprehensive, high-quality proposal that was able to earn the support of the National Science Foundation.”

Top photo: Arizona State University professor Stephen Krause (second from right) is shown meeting with materials science and engineering students to help them develop strategies for making progress on their projects for a capstone engineering design course. Krause emphasizes a teamwork approach in his classes that encourages students to collaborate in acquiring the knowledge and solving the problems necessary to successfully complete assignments and projects. Photographer: Nora Skrodenis/ASU

Joe Kullman

Science writer , Ira A. Fulton Schools of Engineering

480-965-8122 Joseph.Kullman@asu.edu