Top Australian unis to test new science metrics system

The University of Melbourne will oversee a new effort to measure the returns on science investments. Flickr/Geoff Penaluna

The University of Melbourne and Australian National University will perform a “stocktake” of all scientific research projects at the two institutions from the electronic era in an effort to map their social and commercial returns, the head of the project revealed this week.

The study, to be completed next year, will be loosely modelled on the groundbreaking STAR METRICS program, introduced in the US in 2010 to document the outcomes of science investments to the public by measuring their social, industrial and economic impact.

Paul Jensen, Professor of Innovation, Science and Technology Policy at the University of Melbourne, will oversee the scheme. He said he hoped to roll it out to other research institutions in Australia: “It’s definitely time universities looked at the impact of scientific research beyond peer-reviewed publications. To do this, we need to build the appropriate data and develop a more scientific approach to our analysis of science policy.”

Professor Jensen met with STAR METRICS founder Professor Julia Lane in Canberra this week to seek advice about the project, dubbed ASTRA - the Australian Science and Technology Research Assessment.

“I realised that there was a dearth of systematic empirical evidence with which to inform science and technology policy in Australia, that no one knew how to quantify the returns on investment in science,” Professor Jensen said. “So I invited Julia out here as a visiting scholar and started introducing her, and the STAR METRICS platform, to interested stakeholders in the science and technology policy community.

"Out of all of that, we started talking about how we could build out and even extend on the sort of thing they’ve been able to do in the US. We have amazing opportunities to build out because we’re quite small and our investment in science is quite concentrated in a small number of institutions.”

The Federal Government has committed funds to support the feasibility study, the first of its kind in Australia. “We’re hoping it’ll be undertaken in the next six to nine months,” Professor Jensen said. “By early 2013 we’ll be able to present the results, which will include a thorough stocktake on all the research for the two institutions in the program, Australian National University and the University of Melbourne. Both have agreed to participate in a pilot project.”

Although the two institutions are members of the Group of Eight universities, the pilot will be “quite separate” to another metrics trial by the Go8 and Australian Technology Network of Universities, he said. That trial is expected to get underway in coming months.

The logistics of ASTRA are still under discussion, said Professor Jensen, who confirmed he will be bringing IT experts from the STAR METRICS project out to Australia to train staff here. “With their help, we hope to develop capabilities in Australia in the computational skills that are required to build this data platform.

"Once it’s built, down the track we expect that there’ll be a whole lot of social scientists who’ll want to come in and start using the data to do some interesting research and analysis. That’s where the real value add comes. So that we can start providing a better explanation for the sort of returns we’ve been getting for our science investments.”

The pilot will be designed to answer enduring questions in science policy: How much collaboration should universities be promoting? Which programs work, and how do we know? What sort of economic returns do they generate? What is their influence on government or other policy? “Now we just don’t know the answers to any of those important questions.”

“We’re hoping that this will give us a good idea about data issues, and we can figure out how much it would cost to roll it out to other institutions, if they’re willing to take part,” Professor Jensen said.

The pilot will cover the “full quantum” of research at both institutions. “The idea is that rather than taking a snapshot of a particular discipline, we’ll take all the funds that come from, say, the Australian Research Council, and look at how that relates to the quantum of publications, how it relates to the number of patents that are taken out, how it influences policy, and many more things. The idea is that this will be extended to include other impacts of our investment in science.”

How far back the pilot goes will depend on the quality of the data, he said. “If you go back in time you reach a point where scientists didn’t have electronic documents that we can access. The idea is that this will be as low cost as possible. We don’t want scientists wasting time filling out new forms. We can only do this as much as we’re able to electronically compile data, so it depends on the availability of things such as pdf documents and so on.”

Like STAR METRICS, the study will be limited to research in science and technology. This was not to downplay the importance of research in the humanities and social sciences: “It’s just that measuring the outcome of investments in research in those areas is much harder. What we’re proposing here is already hard enough. But eventually we do envisage that those disciplines will be included.”