Heidi Allen Digital Strategy

Thinking about members of my family, and friends, who are in their 50′s and 60′s doing battle (which is how they would view it) with the latest technology,  the questions asked are often about learning to do simple things like email, using search engines, what a blog is (and what is the difference between that and a website?).

This is interesting research from the American Life project run by PEW. For those, like my folks, who are figuring out what the internet is, a smart phone (like an iPhone) would help them as they can be very user friendly.  Instead, they are often too intimidated to try it out.  As it is, according to this research only 22% of Baby Boomers have a smart phone.  And they are, perhaps, those that would benefit a great deal from them.

I first published this article on BodyInMind – a pain research site that I also am involved in.  The tricky question of how we measure, properly measure, the impact in the clinic of disseminating research using social media has come up time and time again in our meetings and as yet we have found no answer.

A recent article in the Journal of Medical Internet Research (JMIR)[1] looked at whether it is feasible to measure social impact of, and public attention to, newly published research articles by analysing buzz in social media – specifically twitter. It also asked whether these metrics are sensitive and specific enough to predict highly cited articles – something that would be valuable for researchers to know.

It might seem a strange thing to use. Twitter is a vehicle for people to communicate to their chosen network, limited to 140 characters per ‘tweet’. How can this chatter be used to predict whether a journal article will be highly cited in the future?

How is research evaluated at the moment? Well at least in two ways. Citations measure the productivity and impact of a researcher, and the impact factor evaluates the impact of a journal. However, citations only measure uptake within the scientific community and take a long time to gather. The impact of research in the real world and uptake by the public is very hard to measure and currently there is no really accurate way of doing it, something which this research hoped to address.

So how was this research done?  Over a period of 3.5 years tweets with links to JMIR were gathered and from these 1600 tweets (or ‘tweetations’[2]) talking about 55 articles in a 2 year period were analyzed. Social media impact was compared against data from Scopus and Google Scholar 17-29 months later (which is how long it takes to gather citations traditionally).

Using this data a new algorithm was devised and tested to see if it was possible to gauge accurately whether an article would be highly cited within one week of publication in JMIR (bearing in mind that this can take up to 2 years to find out).

The author found that if an article is highly tweeted then there is a 75% likelihood that it would end up in the top quartile of all articles of an issue, ranked by citations. Most tweets were sent on the day of publication: 44% of all tweets in a 2 month period, 18% on the following day followed by a rapid decay. In other words, tweets can predict highly cited articles within the first 3 days of article publication. Low impact articles are tweeted and retweeted mainly on day 0 and 1. Highly cited articles continue to be retweeted widely.

The so what factor

We discussed this article as part of our weekly BiM meeting – along with eating some Very Excellent Tiramisu that Luke made – and there are some questions as to bias in this article. The first is that the author, Gunther Eysenbach is founder and editor-in-chief of the JMIR. This journal is open access (freely available) and covers research, information and communication in the healthcare field. As a topic this article is well suited to the journal but it may have been better if it had been peer reviewed and published in another journal, PLoS one perhaps.

The second is that the author is coining new phrases (such as twimpact[3]) introduced as part of his research and has set up websites with that name in the hope, we presume, that the algorithm and metric becomes widely used.[CORRECTION: the Twimpact website is NOT associated with Professor Eysenbach or this research - see his comment on the original article at bodyinmind.org)

There are also some caveats with the research which the author himself points out. Although top cited articles can be predicted from tweeted articles, social impact measures can only complement traditional citation metrics but not replace them.

For example, tweetations are a metric for social impact and how quickly new knowledge is taken up by public, whereas citations are a metric for scholarly impact. They measure uptake by or interest of different audiences. The twimpact factor (cumulative number of tweets after a certain number of days) complements the impact factor in that it is a useful metric to measure uptake of research findings resonating with the public in real time.

At the moment we also don’t know if the twitter mentions are the result of someone influential tweeting and people getting on the popularity bandwagon or if it reflects the actual quality of the article. It only shows us how the question or topic (and possibly conclusions if the article has actually been read) resonates with Twitter. In other words we may be measuring the structure of the network and attributes of social media communities rather than the attributes of the information itself.

Popularity is a useful measure for commercial enterprises but those that do not resonate with the general public, eg low income old age groups, and who are not represented on twitter may lead to further marginalization of these groups.

This is still a very new field and the author (as the editor of JMIR) has issued a standing call for papers to ‘assess the robustness of these social media metrics and their ability to detect signals among the noise of social media chatter’.

He rightly points out that attentiveness to issues is a prerequisite for social change, and tweets are a useful metric to measure attentiveness to a specific scholarly publication. For us at BiM I wonder whether we can use new social media avenues to get the explain pain message out more effectively.  What we can’t yet do is measure what effect, if any, this has at the level of patient care.

Definitions and Reference

[1] Eysenbach, G. (2011). Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact Journal of Medical Internet Research, 13 (4) DOI: 10.2196/jmir.2012

[2] Tweetation – twitter citation eg for seven days tw7. (skewed  by publication date)

[3] twimpact factor – TWIF7 = cumulative number of tweetations 7 days after publication

A little while ago I was interviewed about the work I do at BodyinMind.org.  This is a website that writes about pain research.  We started it just over 2 years ago, and now have over 9,000 unique visitors every month from over 100 countries.  That’s great you might think, but there is still the ‘So What’ factor.  Is this actually making any difference?  The jury is still out and we are trying very hard to measure whether what we are doing makes any difference at all to pain research or clinical practice.

I share an office with James. It was James who alerted me to a fantastic piece of software that even I can understand. Drag the pdf of a paper into papers – it files, sorts, retrieves them with ease. I love it.

Papers Like You’ve Never Seen Before: Papers2.1

Papers – your personal library of research

Here is their blurby bit:

Papers revolutionizes the way you deal with your research documents. It allows you to search for them, download and organize them together with supplementary material, allows you to read them full screen, highlight and keep notes, sync them to your iPad or iPhone, cite them in your favorite word processor, share them with your colleagues, and much much more.

JitterJam combines ‘social media monitoring, an intelligent contact database and a multi-channel digital marketing platform into a single, integrated Social CRM system’.

It is the measuring aspect, rather than the marketing, that I am interested in. Maybe this is a way that I can start looking at whether dissemination of health research has any effect on clinical practice – or are we just having a chat online and not changing clinical practice at all?

I recently looked at whether scientific journals are using social media – particularly medical and pain journals – and presented what I found at the recent Australian Pain Society conference. Here’s a version of that presentation. Before looking at what the study found it might be useful to define what social media is. Wikipedia came up with:

‘Social media are media for social interaction, using highly accessible and scalable communication techniques. It is the use of web-based and mobile technologies to turn communication into interactive dialogue’

What does that mean? I would suggest social media is a way to reach a specific audience, like researchers and clinicians with targeted information on a variety of platforms such as twitter, or facebook or youtube, or blogs through different devices (laptop, mobile phones, tablets).  Why is this relevant to research journals?

A recent paper in Nature talks about the fact that researchers are undergoing trial by twitter ‘Blogs and tweets are ripping papers apart within days of publication, leaving researchers unsure how to react.’ It potentially represents shift in the way we communicate and disseminate research. It lead me to wonder whether Scientific journals are using social media? Does social media have any relevance for dissemination of research and clinical practice?

The background to this study is that we know that the application of evidence based medicine to clinical practice obligates us to keep up to date with research progress. Journals are the most important means of doing that, but rely on their audience ‘pulling’ relevant findings from the wider pool of literature.  The role of social media in connecting information to users presents an opportunity for journals to ‘push’ the relevant findings to their audience in a targeted and time-efficient manner.

The aim of this study was to determine the use of social media by medical and pain  journals to ascertain whether they had a SM presence, if they did, how they used it, (participation), and whether they hand any influence.  What did we do?  We selected the top 10 general medical and the top six pain-focussed journals according to their impact factor. Whether those journals had a social media presence of a blog, facebook, YouTube or twitter streams was determined by Google search as well as searching on Journal websites individually, and separately on facebook, youtube and twitter.

Why did we chose these platforms? Nielsen indicates that facebook, youtube and  twitter are the three main social media platforms used, and these are often linked to a blog.  For participation in social media we counted the number of posts over a one month period (Jan 2011) and for influence we used hubspot’s twitter and facebook grader to provide a measure of how the social media presence might be ranked.  We did not have an algorithm to measure YouTube influence and simply noted presence for this social media site.

What did we find? Several high-impact general medical journals had a strong social media presence, 9 out of the 10 had twitter accounts that were regularly participated in.  For example the BMJ has a twitter account, and posted 81 times in the month of January and has 97.6 percent influence.  Influence was measured in a number of ways based on number of posts, interactions, number of times the information was shared as well as follower/following ratio.

What else did we find?  Eight out of 10 top medical journals had facebook accounts and 5 out of 10 had a blog, but only 3 had a YouTube account.  Looking at the data it would be interesting to do a correlation to see if a higher impact factor corresponds with greater social media presence.

What did we find with pain journals? One brave soul – Molecular pain, had a twitter account.  However, I would be hesitant in calling this a presence, on closer examination of their twitter account they have no logo, no url to direct followers back to the website and only one post in January (the second was in April – so clearly there is no dialogue or regular posting to followers). In contrast you can see some the medical journals with an active twitter presence – in this case here you can see NEJM, the Lancet and PLoS on twitter, facebook and blog pages.

What can we conclude? The top general medical journals use social media to disseminate their content. By so doing, they push their information to thousands of audience members each week. In contrast, pain-focussed journals have no social media presence and as such miss out on this opportunity.

However, the impact of social media-based dissemination on clinical practice remains to be investigated, but these preliminary data suggest that social media could be better used to disseminate new findings in the pain sciences.