I first published this article on BodyInMind – a pain research site that I also am involved in. The tricky question of how we measure, properly measure, the impact in the clinic of disseminating research using social media has come up time and time again in our meetings and as yet we have found no answer.
A recent article in the Journal of Medical Internet Research (JMIR)[1] looked at whether it is feasible to measure social impact of, and public attention to, newly published research articles by analysing buzz in social media – specifically twitter. It also asked whether these metrics are sensitive and specific enough to predict highly cited articles – something that would be valuable for researchers to know.
It might seem a strange thing to use. Twitter is a vehicle for people to communicate to their chosen network, limited to 140 characters per ‘tweet’. How can this chatter be used to predict whether a journal article will be highly cited in the future?
How is research evaluated at the moment? Well at least in two ways. Citations measure the productivity and impact of a researcher, and the impact factor evaluates the impact of a journal. However, citations only measure uptake within the scientific community and take a long time to gather. The impact of research in the real world and uptake by the public is very hard to measure and currently there is no really accurate way of doing it, something which this research hoped to address.
So how was this research done? Over a period of 3.5 years tweets with links to JMIR were gathered and from these 1600 tweets (or ‘tweetations’[2]) talking about 55 articles in a 2 year period were analyzed. Social media impact was compared against data from Scopus and Google Scholar 17-29 months later (which is how long it takes to gather citations traditionally).
Using this data a new algorithm was devised and tested to see if it was possible to gauge accurately whether an article would be highly cited within one week of publication in JMIR (bearing in mind that this can take up to 2 years to find out).
The author found that if an article is highly tweeted then there is a 75% likelihood that it would end up in the top quartile of all articles of an issue, ranked by citations. Most tweets were sent on the day of publication: 44% of all tweets in a 2 month period, 18% on the following day followed by a rapid decay. In other words, tweets can predict highly cited articles within the first 3 days of article publication. Low impact articles are tweeted and retweeted mainly on day 0 and 1. Highly cited articles continue to be retweeted widely.
The so what factor
We discussed this article as part of our weekly BiM meeting – along with eating some Very Excellent Tiramisu that Luke made – and there are some questions as to bias in this article. The first is that the author, Gunther Eysenbach is founder and editor-in-chief of the JMIR. This journal is open access (freely available) and covers research, information and communication in the healthcare field. As a topic this article is well suited to the journal but it may have been better if it had been peer reviewed and published in another journal, PLoS one perhaps.
The second is that the author is coining new phrases (such as twimpact[3]) introduced as part of his research and has set up websites with that name in the hope, we presume, that the algorithm and metric becomes widely used.[CORRECTION: the Twimpact website is NOT associated with Professor Eysenbach or this research - see his comment on the original article at bodyinmind.org)
There are also some caveats with the research which the author himself points out. Although top cited articles can be predicted from tweeted articles, social impact measures can only complement traditional citation metrics but not replace them.
For example, tweetations are a metric for social impact and how quickly new knowledge is taken up by public, whereas citations are a metric for scholarly impact. They measure uptake by or interest of different audiences. The twimpact factor (cumulative number of tweets after a certain number of days) complements the impact factor in that it is a useful metric to measure uptake of research findings resonating with the public in real time.
At the moment we also don’t know if the twitter mentions are the result of someone influential tweeting and people getting on the popularity bandwagon or if it reflects the actual quality of the article. It only shows us how the question or topic (and possibly conclusions if the article has actually been read) resonates with Twitter. In other words we may be measuring the structure of the network and attributes of social media communities rather than the attributes of the information itself.
Popularity is a useful measure for commercial enterprises but those that do not resonate with the general public, eg low income old age groups, and who are not represented on twitter may lead to further marginalization of these groups.
This is still a very new field and the author (as the editor of JMIR) has issued a standing call for papers to ‘assess the robustness of these social media metrics and their ability to detect signals among the noise of social media chatter’.
He rightly points out that attentiveness to issues is a prerequisite for social change, and tweets are a useful metric to measure attentiveness to a specific scholarly publication. For us at BiM I wonder whether we can use new social media avenues to get the explain pain message out more effectively. What we can’t yet do is measure what effect, if any, this has at the level of patient care.
Definitions and Reference
[1] Eysenbach, G. (2011). Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact Journal of Medical Internet Research, 13 (4) DOI: 10.2196/jmir.2012
[2] Tweetation – twitter citation eg for seven days tw7. (skewed by publication date)
[3] twimpact factor – TWIF7 = cumulative number of tweetations 7 days after publication
