The art of the systematic review with Leila Ledbetter

1 00:00:00,000 --> 00:00:05,860 Welcome to the Clinician Researcher podcast, where academic clinicians learn the skills 2 00:00:05,860 --> 00:00:11,260 to build their own research program, whether or not they have a mentor. 3 00:00:11,260 --> 00:00:17,340 As clinicians, we spend a decade or more as trainees learning to take care of patients. 4 00:00:17,340 --> 00:00:22,380 When we finally start our careers, we want to build research programs, but then we find 5 00:00:22,380 --> 00:00:27,780 that our years of clinical training did not adequately prepare us to lead our research 6 00:00:27,780 --> 00:00:29,200 program. 7 00:00:29,200 --> 00:00:35,480 Through no fault of our own, we struggle to find mentors, and when we can't, we quit. 8 00:00:35,480 --> 00:00:40,580 However, clinicians hold the keys to the greatest research breakthroughs. 9 00:00:40,580 --> 00:00:46,200 For this reason, the Clinician Researcher podcast exists to give academic clinicians 10 00:00:46,200 --> 00:00:51,800 the tools to build their own research program, whether or not they have a mentor. 11 00:00:51,800 --> 00:01:01,100 Now introducing your host, Toyosi Onwuemene. 12 00:01:01,100 --> 00:01:03,060 Welcome to the Clinician Researcher podcast. 13 00:01:03,060 --> 00:01:07,220 I'm your host, Toyosi Onwuemene, and I'm super excited to be talking with you today because 14 00:01:07,220 --> 00:01:08,220 I have a special guest. 15 00:01:08,220 --> 00:01:10,980 Actually, Leila is not new to the show. 16 00:01:10,980 --> 00:01:12,420 She's been here before. 17 00:01:12,420 --> 00:01:18,700 She is a wonderful and amazing medical information specialist, also known as a librarian. 18 00:01:18,700 --> 00:01:23,700 And she in particular is here today because of her expertise in systematic reviews. 19 00:01:23,700 --> 00:01:29,740 In fact, she is chair of the Systematic Review Caucus for the Medical Library Association. 20 00:01:29,740 --> 00:01:32,700 And so she's here to talk to us today about systematic reviews. 21 00:01:32,700 --> 00:01:34,260 Leila, welcome to the show. 22 00:01:34,260 --> 00:01:36,240 Thank you so much for having me again. 23 00:01:36,240 --> 00:01:40,420 This is super fun, and I love talking about systematic reviews. 24 00:01:40,420 --> 00:01:43,600 Well, systematic reviews is definitely what we're talking about today. 25 00:01:43,600 --> 00:01:48,420 And I'm so glad that you're here because sometimes it seems as if it should be intuitive. 26 00:01:48,420 --> 00:01:50,220 You're doing a systematic review. 27 00:01:50,220 --> 00:01:53,780 Just search the literature, pull the papers together, write the paper. 28 00:01:53,780 --> 00:01:57,700 It seems like it should be straightforward, but it's a lot more complicated than that. 29 00:01:57,700 --> 00:02:03,380 So I will start by saying when we first started working together, I had no idea about systematic 30 00:02:03,380 --> 00:02:05,300 reviews and how long it would take. 31 00:02:05,300 --> 00:02:08,820 My first thought when someone told me that it would take about six months to do a good 32 00:02:08,820 --> 00:02:13,580 systematic review was to laugh and say, why would anybody need so much time? 33 00:02:13,580 --> 00:02:18,220 I feel like a little bit wiser now, but I want you to just talk about what does it take 34 00:02:18,220 --> 00:02:24,460 to do a good systematic review, not just in terms of the time, the effort, but also the 35 00:02:24,460 --> 00:02:28,020 resources that you need in personnel and all things. 36 00:02:28,020 --> 00:02:29,020 Okay. 37 00:02:29,020 --> 00:02:30,020 Wow. 38 00:02:30,020 --> 00:02:31,020 This is a big question. 39 00:02:31,020 --> 00:02:35,260 So you may have to refocus me as we cruise through the answer to that. 40 00:02:35,260 --> 00:02:37,540 So maybe we can break it up into parts. 41 00:02:37,540 --> 00:02:44,300 So we kind of joke that systematic reviews are one of the research methods that nobody 42 00:02:44,300 --> 00:02:47,740 seems to go to the manual for. 43 00:02:47,740 --> 00:02:53,340 There's manuals and checklists and things to tell you how to do them, but nobody does 44 00:02:53,340 --> 00:02:54,340 that. 45 00:02:54,340 --> 00:02:57,620 They just decide that they're just going to launch themselves in. 46 00:02:57,620 --> 00:03:02,580 And then as they sort of cruise down the road, we're doing things, they're surprised by what's 47 00:03:02,580 --> 00:03:04,540 involved in it. 48 00:03:04,540 --> 00:03:10,100 So some of the things that we say, and you remember this at our first consultation where 49 00:03:10,100 --> 00:03:17,100 we sit down with a new team is we ask people how many people are on their team. 50 00:03:17,100 --> 00:03:23,660 So we encourage the minimum, minimum, minimum would be two people, but we encourage you 51 00:03:23,660 --> 00:03:28,820 to include a librarian or team to help you with your searches because you would prefer 52 00:03:28,820 --> 00:03:31,180 to have an expert do that than to do it yourself. 53 00:03:31,180 --> 00:03:35,820 It really ends up, we can talk about that a little bit later, ends up being very complicated. 54 00:03:35,820 --> 00:03:42,180 You're going to need, and we usually say six months at the minimum for an experienced team 55 00:03:42,180 --> 00:03:47,860 with a really tight question and enough like hands on deck because right, you could end 56 00:03:47,860 --> 00:03:51,900 up with fed teams of like 18,000 things to screen. 57 00:03:51,900 --> 00:03:56,780 So on an average, we actually say it takes about 12 months and on the outside, kind of 58 00:03:56,780 --> 00:03:59,480 like 18 is not unusual. 59 00:03:59,480 --> 00:04:03,500 So that's because you have to figure out what your question is. 60 00:04:03,500 --> 00:04:05,700 You have to do the searches. 61 00:04:05,700 --> 00:04:09,820 You might end up having to refine your question depending on how big your question is. 62 00:04:09,820 --> 00:04:16,880 It might be, you know, like 18,000 things and that's about 200 things an hour. 63 00:04:16,880 --> 00:04:20,500 So if you divvy that up amongst your right, it could take months and months and months 64 00:04:20,500 --> 00:04:22,980 just to get through your screening. 65 00:04:22,980 --> 00:04:27,340 Then you have to read the papers and then you have to, and you probably know this, but 66 00:04:27,340 --> 00:04:31,860 nobody thinks about it is when you get to where you actually have to pull the data from 67 00:04:31,860 --> 00:04:33,500 the papers you've selected. 68 00:04:33,500 --> 00:04:36,500 That's the most time consuming piece. 69 00:04:36,500 --> 00:04:40,180 And I feel like we should spend more time talking about like, what are you going to 70 00:04:40,180 --> 00:04:42,360 collect and how are you going to collect it? 71 00:04:42,360 --> 00:04:44,220 How do you plan to normalize your data? 72 00:04:44,220 --> 00:04:46,060 And so like all these things. 73 00:04:46,060 --> 00:04:52,180 And in the meantime, you've got software and you have manuals and you have reporting guidelines 74 00:04:52,180 --> 00:04:56,340 of things that you have to meet in order to publish the work that you've done. 75 00:04:56,340 --> 00:04:59,260 There's just so many things in that whole process. 76 00:04:59,260 --> 00:05:00,260 Is that a good overview? 77 00:05:00,260 --> 00:05:01,260 Yes. 78 00:05:01,260 --> 00:05:02,480 No, it's good. 79 00:05:02,480 --> 00:05:06,940 It's good because what I'm hearing you say, and I'm so glad that you're communicating 80 00:05:06,940 --> 00:05:11,620 to the audience is that this is not just a walk in the park. 81 00:05:11,620 --> 00:05:14,980 This is not just a pull a couple of papers together and do a review. 82 00:05:14,980 --> 00:05:16,580 It really is systematic. 83 00:05:16,580 --> 00:05:20,140 There's a process to it and it takes time and it takes energy. 84 00:05:20,140 --> 00:05:22,340 It takes people to do it as well. 85 00:05:22,340 --> 00:05:26,820 So speaking about people, your librarian, why should you involve your librarian in the 86 00:05:26,820 --> 00:05:27,820 systematic review? 87 00:05:27,820 --> 00:05:32,180 Aren't we just putting a few keywords into PubMed and finding a couple of papers? 88 00:05:32,180 --> 00:05:33,180 I love this. 89 00:05:33,180 --> 00:05:36,540 I mean, you can, but what we usually warn people. 90 00:05:36,540 --> 00:05:41,540 So a systematic review is a very defined methodology. 91 00:05:41,540 --> 00:05:47,140 And one of the reasons it is at the highest level of the pyramid of your evidence pyramid 92 00:05:47,140 --> 00:05:52,760 is that what you've said is we have gone out there and found all of the studies about this 93 00:05:52,760 --> 00:05:56,980 particular topic and we've done things to them. 94 00:05:56,980 --> 00:06:02,160 One of the things that also defines a systematic review is evaluating the quality of the article 95 00:06:02,160 --> 00:06:04,420 so we can talk about that a little bit later. 96 00:06:04,420 --> 00:06:10,100 But you go out and you have to find all of that and it's really embarrassing if you go 97 00:06:10,100 --> 00:06:13,220 to publish something and you have not found it. 98 00:06:13,220 --> 00:06:20,420 It's also a little bit dangerous because your audience trusts that you have found everything 99 00:06:20,420 --> 00:06:25,460 and that the recommendation you are giving with your paper is accurate because you have 100 00:06:25,460 --> 00:06:26,900 found everything. 101 00:06:26,900 --> 00:06:32,860 And you have worked with me and with other librarians and we're just used to, just like 102 00:06:32,860 --> 00:06:38,020 clinicians are, we're used to thinking about the problem within our scope of expertise 103 00:06:38,020 --> 00:06:40,500 in a really specific way. 104 00:06:40,500 --> 00:06:43,940 So we're used to thinking about the other ways topics are talked about. 105 00:06:43,940 --> 00:06:49,180 We're really, we have to do at least three databases for a systematic review so we know 106 00:06:49,180 --> 00:06:52,140 the language and syntax for each one. 107 00:06:52,140 --> 00:06:57,020 We know how to pull back enough stuff but not more than you can handle. 108 00:06:57,020 --> 00:07:01,140 There's all these little tips and tricks that come just with the expertise of searching 109 00:07:01,140 --> 00:07:02,780 for information. 110 00:07:02,780 --> 00:07:06,180 And that's your data that you base the rest of your paper on. 111 00:07:06,180 --> 00:07:11,700 So all the methodologies and stuff say we really recommend working with some sort of 112 00:07:11,700 --> 00:07:17,420 informationist in order to get the data that you then work on for the paper. 113 00:07:17,420 --> 00:07:19,780 So we have to collect your data. 114 00:07:19,780 --> 00:07:21,660 Yeah, that's awesome. 115 00:07:21,660 --> 00:07:26,260 So you talk about the need to be systematic. 116 00:07:26,260 --> 00:07:28,820 So this is a research methodology. 117 00:07:28,820 --> 00:07:34,260 Systematically searching the literature is a unique skill set. 118 00:07:34,260 --> 00:07:38,800 It takes expertise to be able to do it well and make sure that you're systematically capturing 119 00:07:38,800 --> 00:07:42,200 all articles on that topic. 120 00:07:42,200 --> 00:07:47,100 And so in the same way as clinicians we spend a long time trying to figure out how to do 121 00:07:47,100 --> 00:07:48,580 our clinical specialty. 122 00:07:48,580 --> 00:07:53,860 Well, as librarians you take time figuring out how to do these systematic searches to 123 00:07:53,860 --> 00:07:55,820 the best of your ability. 124 00:07:55,820 --> 00:08:01,660 Yes, and I could teach someone if they wanted to spend the time practicing just like any 125 00:08:01,660 --> 00:08:08,340 sport or skill if you had the time and the opportunity to practice then you could be 126 00:08:08,340 --> 00:08:10,300 just as good at it. 127 00:08:10,300 --> 00:08:18,420 It really does take months to years to really learn all the tips and tricks and skills to 128 00:08:18,420 --> 00:08:19,420 end up doing that. 129 00:08:19,420 --> 00:08:24,260 And I continue to learn things from other experts who have figured stuff out. 130 00:08:24,260 --> 00:08:27,220 And we're just like you guys in a way. 131 00:08:27,220 --> 00:08:29,020 It's a clinical skill. 132 00:08:29,020 --> 00:08:31,060 Absolutely, absolutely. 133 00:08:31,060 --> 00:08:36,060 So we started with team and as part of your team you want at least two people but probably 134 00:08:36,060 --> 00:08:41,460 more and your librarian needs to be part of that team so that you can do a good systematic 135 00:08:41,460 --> 00:08:42,460 search. 136 00:08:42,460 --> 00:08:43,460 So we've got the team assembled. 137 00:08:43,460 --> 00:08:45,860 Now let's talk about the protocol. 138 00:08:45,860 --> 00:08:48,780 Why do you need to write a protocol before you get started? 139 00:08:48,780 --> 00:08:51,980 Why not just define the question and go? 140 00:08:51,980 --> 00:08:59,860 Yeah, so I liken it to making a decision about where you go before you start the car and 141 00:08:59,860 --> 00:09:01,460 driving, right? 142 00:09:01,460 --> 00:09:07,940 So you type in your GPS and if you're smart you'll have a quick look to make sure it's 143 00:09:07,940 --> 00:09:11,380 taking you someplace that makes sense, right? 144 00:09:11,380 --> 00:09:15,540 Instead of just being like okay and then getting to where the bridge is out. 145 00:09:15,540 --> 00:09:16,860 How did I get here? 146 00:09:16,860 --> 00:09:20,540 Or the traffic's terrible or whatever, right? 147 00:09:20,540 --> 00:09:26,780 So a protocol is your roadmap or your planning document for your project. 148 00:09:26,780 --> 00:09:31,980 So just like you would do a clinical trial, any kind of experiment you would set up how 149 00:09:31,980 --> 00:09:34,700 you plan, execute all of those things. 150 00:09:34,700 --> 00:09:41,260 So then you have to stop every time you hit a place and say now what do we do? 151 00:09:41,260 --> 00:09:43,180 Okay we've done the search, now what? 152 00:09:43,180 --> 00:09:46,620 Okay now we've loaded it in the software, okay now what? 153 00:09:46,620 --> 00:09:49,140 All right now we're onto the next step, right? 154 00:09:49,140 --> 00:09:54,460 So you end up if you don't have a protocol kind of coming with screeching stop and it 155 00:09:54,460 --> 00:10:01,420 really slows down your flow and I just feel like kind of knowing where you're going, what 156 00:10:01,420 --> 00:10:07,860 data you plan to collect, how you plan to evaluate the articles, all of that makes you 157 00:10:07,860 --> 00:10:10,440 actually better at the earlier steps. 158 00:10:10,440 --> 00:10:15,460 So really having a good idea of your final destination and that's not just well I plan 159 00:10:15,460 --> 00:10:21,540 to do X but like the things that you plan to collect and the way you plan to do it really 160 00:10:21,540 --> 00:10:24,220 makes a huge difference early on in the process. 161 00:10:24,220 --> 00:10:28,100 Have you found that to be the case now that you know where you're going? 162 00:10:28,100 --> 00:10:34,780 So your protocol is that roadmap and also it's a little bit to keep you from cheating. 163 00:10:34,780 --> 00:10:38,940 I mean this in the like nice way, just something you can't just as you're cruising along and 164 00:10:38,940 --> 00:10:45,540 you realize this is not going the way I expected it to which is like 50% of science maybe more. 165 00:10:45,540 --> 00:10:50,820 You can't just pivot, it's supposed to keep you honest, you know so this is what I said 166 00:10:50,820 --> 00:10:56,380 I was going to do and because it's not working out the way I wanted it to you can either 167 00:10:56,380 --> 00:11:01,060 quit and just opt not to but you can't cheat and change it, right? 168 00:11:01,060 --> 00:11:02,940 The protocol keeps you honest. 169 00:11:02,940 --> 00:11:08,340 Absolutely, absolutely and I love the what you say about it being the roadmap. 170 00:11:08,340 --> 00:11:11,340 What it allows you to do because you're answering these questions as you're going through the 171 00:11:11,340 --> 00:11:15,900 protocol it allows you to think early about challenges that you might face and address 172 00:11:15,900 --> 00:11:19,540 and decide how you're going to address them before you get to those challenges so it's 173 00:11:19,540 --> 00:11:21,820 definitely very helpful. 174 00:11:21,820 --> 00:11:26,500 Now you can't just you so you put your protocol together but you also have to register it. 175 00:11:26,500 --> 00:11:28,380 What is this thing about registering protocols? 176 00:11:28,380 --> 00:11:30,380 Can you speak to that? 177 00:11:30,380 --> 00:11:36,860 Yes, so there are a protocol is also helpful because it helps you plant your flag in your 178 00:11:36,860 --> 00:11:42,900 topic, it lets people know hey don't even bother we've already started working on this 179 00:11:42,900 --> 00:11:47,980 here's our question here's what we plan to search and so you can go in and look at these 180 00:11:47,980 --> 00:11:54,620 registries and say oh Toyosi is working on this already I'm going to find another aspect 181 00:11:54,620 --> 00:11:59,860 of this topic that I'm particularly interested in so it saves you from wasting any time on 182 00:11:59,860 --> 00:12:02,540 a topic somebody else is already working on. 183 00:12:02,540 --> 00:12:08,060 It also saves you from having like a competing paper out there it's just that's better all 184 00:12:08,060 --> 00:12:14,720 around so the registries are a little bit like registering a car I mean there's nobody 185 00:12:14,720 --> 00:12:22,540 really policing the quality of the protocols there is a organization called Prospero they 186 00:12:22,540 --> 00:12:28,100 are out of the University of York in the UK and they will give you like a registration 187 00:12:28,100 --> 00:12:35,500 number for your protocol and they do do like a quick look through to make sure that everything 188 00:12:35,500 --> 00:12:41,020 looks legit you know did you talk about what kind of data you're collecting did you say 189 00:12:41,020 --> 00:12:46,140 you were going to search certain number of databases it doesn't mean your systematic 190 00:12:46,140 --> 00:12:50,340 review is going to be any good but they are checking to make sure you fit all the high 191 00:12:50,340 --> 00:12:56,180 points so there is Prospero they only accept systematic review registration so there are 192 00:12:56,180 --> 00:13:02,500 other evidence syntheses like scoping reviews rapid reviews there's a ton of other types 193 00:13:02,500 --> 00:13:08,860 of evidence syntheses and those will go into another registry the most common one I use 194 00:13:08,860 --> 00:13:15,460 is the OSF which is the Open Science Foundation I believe I can never remember what the F 195 00:13:15,460 --> 00:13:24,420 stands for but they also accept registration and will give you a number and a DOI and copyright 196 00:13:24,420 --> 00:13:31,340 for it and all that good stuff so that is a way for you to say look here is my a priori 197 00:13:31,340 --> 00:13:38,020 project this is how I planned to conduct it we didn't talk about another thing that protocols 198 00:13:38,020 --> 00:13:45,140 do which is a lot of journals now require protocols if you're nodding your head I actually 199 00:13:45,140 --> 00:13:51,180 had some poor team say that they tried to get their manuscript published and they hadn't 200 00:13:51,180 --> 00:13:58,140 registered a protocol and the journal that they wanted to publish in was like no you 201 00:13:58,140 --> 00:14:02,940 needed that to do one can we do one now and I'm like no that has to be done ahead of time 202 00:14:02,940 --> 00:14:06,220 or you're out of luck you're just going to have to find another journal and that's happened 203 00:14:06,220 --> 00:14:11,060 enough times now that I try to warn people because it seems annoying right you know why 204 00:14:11,060 --> 00:14:16,860 should I just like to get on with it but yeah so a lot of journals do it keeps you honest 205 00:14:16,860 --> 00:14:22,060 keeps you on track lets you plan things out it helps you with plant your flag on your 206 00:14:22,060 --> 00:14:27,740 topic yeah they're terrific I'm a huge proponent of doing them absolutely and my experience 207 00:14:27,740 --> 00:14:32,440 they also give you feedback they'll say well think again about your analysis you haven't 208 00:14:32,440 --> 00:14:37,740 included X Y Z so it actually is is helpful feedback so it's definitely the turnaround 209 00:14:37,740 --> 00:14:41,580 gosh I feel like is a couple of weeks to months depending on how busy they are they will get 210 00:14:41,580 --> 00:14:47,940 feedback that's really nice I have not heard you've gotten the most useful feedback I've 211 00:14:47,940 --> 00:14:53,220 heard a lot of times they're just like this is not a good question and you know and zoom 212 00:14:53,220 --> 00:14:58,020 it out but it sounds like you got some really I like hearing that yeah yeah I would say 213 00:14:58,020 --> 00:15:02,980 that one experience we've had is where I think they rejected the protocol and then when we 214 00:15:02,980 --> 00:15:07,140 asked and said what was the problem then they listed the issues and we were able to go back 215 00:15:07,140 --> 00:15:12,420 and fix those which is great that's fabulous I'm gonna let my teams know that that I had 216 00:15:12,420 --> 00:15:18,940 not heard that so that's great there is actually a like a reporting that there are actually 217 00:15:18,940 --> 00:15:23,780 resources out there to help you do a protocol so this again and your librarian will help 218 00:15:23,780 --> 00:15:29,140 you again you don't have to go into this not knowing what's going on I heard from another 219 00:15:29,140 --> 00:15:34,140 clinician that I work with we were asking him you know like do you do protocols now 220 00:15:34,140 --> 00:15:41,140 what is a barrier to this and he said that if as a brand new person to systematic reviews 221 00:15:41,140 --> 00:15:46,500 I had asked him to do a protocol he would have that would have been just too big of 222 00:15:46,500 --> 00:15:51,220 a hump to get over that that was just too scary but I'd like to tell people in your 223 00:15:51,220 --> 00:15:56,700 audience that it actually isn't that big of a deal and maybe you can reassure them it's 224 00:15:56,700 --> 00:16:01,340 literally just kind of like an outline with things that you've decisions you've made and 225 00:16:01,340 --> 00:16:06,100 thoughts you've had about your project did you want to talk about that a little bit absolutely 226 00:16:06,100 --> 00:16:11,180 I will say that the most helpful resource I found which I always go back to and our 227 00:16:11,180 --> 00:16:16,180 library website is where there's a systematic review protocol template and it's a series 228 00:16:16,180 --> 00:16:22,060 of questions and you just answer the questions and it makes it so easy it helps you think 229 00:16:22,060 --> 00:16:25,500 about every aspect so you're not really creating a protocol from scratch as much as answering 230 00:16:25,500 --> 00:16:29,500 a series of questions and that becomes your protocol template so that's that's been really 231 00:16:29,500 --> 00:16:35,100 helpful yes and it doesn't it's like a couple of hours right it's not days and days or months 232 00:16:35,100 --> 00:16:40,940 or months right absolutely absolutely so now when I work with trainees who are working 233 00:16:40,940 --> 00:16:45,060 on systematic reviews that's the first thing I give them is here's the template go fill 234 00:16:45,060 --> 00:16:49,620 this out and then we have conversations around it which is great yeah it's it's really I 235 00:16:49,620 --> 00:16:53,900 love hearing you talk about it because it's sometimes a struggle to get teams to want 236 00:16:53,900 --> 00:16:58,500 to do it again it's the reading the manual there's the people who read the manual and 237 00:16:58,500 --> 00:17:05,940 the people who do not and and I know about this because you taught me well let's talk 238 00:17:05,940 --> 00:17:09,540 about let's talk about the conversation with the librarian when you say I want to do a 239 00:17:09,540 --> 00:17:14,260 systematic review and I have this protocol what information do you need to get to your 240 00:17:14,260 --> 00:17:20,380 librarian other than the protocol we don't always actually require the protocol from 241 00:17:20,380 --> 00:17:25,020 the get-go it's nice if you do if you've done that work ahead of time and you have lovely 242 00:17:25,020 --> 00:17:29,660 teens that almost always do this so but a lot of people just arrive on my doorstep but 243 00:17:29,660 --> 00:17:33,620 they're like little bag that says you know I want to do a systematic review we almost 244 00:17:33,620 --> 00:17:42,060 always talk about their question so that is probably it you might agree with this probably 245 00:17:42,060 --> 00:17:48,700 the biggest struggle is really refining your question I always warn people you're not writing 246 00:17:48,700 --> 00:17:54,940 a textbook this isn't the Moby Dick of systematic reviews you have to include all the things 247 00:17:54,940 --> 00:18:00,980 in fact it's actually better at least in my opinion is these like sports car systematic 248 00:18:00,980 --> 00:18:06,140 reviews small and fast so that you can then have another one and you can get that published 249 00:18:06,140 --> 00:18:10,700 right you can get it done in six to twelve months get that published you can start working 250 00:18:10,700 --> 00:18:15,780 on another one but the people who want to do these like big warship of systematic reviews 251 00:18:15,780 --> 00:18:19,580 I'm like oh it's gonna take you forever and by the time you finish that somebody else 252 00:18:19,580 --> 00:18:25,340 will have published it's full it's let's try to keep it fast like a sports car so we talk 253 00:18:25,340 --> 00:18:30,740 to them about their question and like find out what's how much information is out there 254 00:18:30,740 --> 00:18:35,740 should we make the question bigger sometimes we have to where do we have to make it smaller 255 00:18:35,740 --> 00:18:44,660 we ask them for what papers they already know exist for their topic that is exceptionally 256 00:18:44,660 --> 00:18:51,180 helpful for a lot of reasons one it lets the librarian we use it a lot we use it to sort 257 00:18:51,180 --> 00:18:57,100 of get to know your topic we use it to gather search terms both those mesh terms that everybody's 258 00:18:57,100 --> 00:19:02,660 always talking about the subject headings and also other ways your topic is talked about 259 00:19:02,660 --> 00:19:08,380 it lets us check our searches so we can make sure that the articles you know you want to 260 00:19:08,380 --> 00:19:13,540 include in your systematic review are and pulled up in the search there's just so many 261 00:19:13,540 --> 00:19:19,540 ways we can use it it's very very helpful so we'll ask for the question we'll ask for 262 00:19:19,540 --> 00:19:26,340 those example article maybe like exemplars or what I usually tell people is articles 263 00:19:26,340 --> 00:19:32,820 you expect to include in your in your final set and more than five would be helpful but 264 00:19:32,820 --> 00:19:36,540 if you've got three you've got three did you know that you can publish a systematic review 265 00:19:36,540 --> 00:19:43,500 with nothing in it I mean more I know isn't that interesting so we just published one 266 00:19:43,500 --> 00:19:52,180 you can this usually happens when a team is asked to do a review on a particular topic 267 00:19:52,180 --> 00:19:56,820 and then they go digging and they don't find anything that meets their criteria and so 268 00:19:56,820 --> 00:20:03,340 it ends up being a empty review what this particular team just recently happened did 269 00:20:03,340 --> 00:20:08,980 is they it was a mental health one so they used it as a call to action look nobody is 270 00:20:08,980 --> 00:20:18,140 doing this work we should be you know developing studies and testing if this if this works 271 00:20:18,140 --> 00:20:25,580 if this intervention works so that was fascinating so don't be scared off and I've also had 272 00:20:25,580 --> 00:20:32,060 a review with I had three articles in it don't be scared off by a small amount of of reviews 273 00:20:32,060 --> 00:20:35,580 now it has to be an interesting enough topic that people be interested there's nothing 274 00:20:35,580 --> 00:20:43,820 like that was if you know so what about salamanders I guess you know but that particular thing 275 00:20:43,820 --> 00:20:50,620 was really really I thought a glaring hole in the research and so they did get that published 276 00:20:50,620 --> 00:20:56,220 and it was very interesting so yes you can have zero and I would not encourage you to 277 00:20:56,220 --> 00:21:01,060 do more than how many of you have you had a lot in your reviews if you had like more 278 00:21:01,060 --> 00:21:07,780 than 50 rarely rarely I think that I feel like the numbers in my head are anywhere from 279 00:21:07,780 --> 00:21:14,100 11 to probably the most you've had is 32 yeah I try to encourage people to like stay under 280 00:21:14,100 --> 00:21:19,620 the 50s if you're like there are hundreds of papers out there about this particular 281 00:21:19,620 --> 00:21:24,420 question then I would usually tell people you might want to break your question we might 282 00:21:24,420 --> 00:21:29,420 want to focus more part of that is you're going to have to mention let's say there's 283 00:21:29,420 --> 00:21:35,540 a 250 papers this has happened to one of my teams we have 276 papers they said in our 284 00:21:35,540 --> 00:21:42,100 final set and I'm like you're gonna have to write about all 276 papers in your manuscript 285 00:21:42,100 --> 00:21:46,100 that's gonna have to go in a table in your manuscript you're gonna have to quality assess 286 00:21:46,100 --> 00:21:50,220 all 276 we're gonna have a data extract it that's gonna have to go in your references 287 00:21:50,220 --> 00:21:55,300 where your journal even allow you to have 276 things in your references and they're 288 00:21:55,300 --> 00:22:02,860 like oh so we talked about like what you could do to narrow their topic and one of the things 289 00:22:02,860 --> 00:22:06,980 they did was they actually decided on their follow-up time for this particular thing and 290 00:22:06,980 --> 00:22:11,820 once they decided on a narrower follow-up time that scooched right down to 35 papers 291 00:22:11,820 --> 00:22:16,820 which is perfect in my opinion that is great so one of the things I hear you talking about 292 00:22:16,820 --> 00:22:21,160 is really a librarian as your partner in this process not not just the person who comes 293 00:22:21,160 --> 00:22:25,860 and searches and then it's done but really a partner and so in thinking about the partnership 294 00:22:25,860 --> 00:22:31,860 so you help with the search your co-authors run off and they do the title and abstract 295 00:22:31,860 --> 00:22:36,860 screening and then they're like wow we found these maybe 72 articles that we think meet 296 00:22:36,860 --> 00:22:41,900 inclusion criteria and now we actually have to go find the articles how are you helpful 297 00:22:41,900 --> 00:22:49,220 at that point so one of the things that we can do is we can use software to help pull 298 00:22:49,220 --> 00:22:55,540 articles for that for your manuscripts so let's say there's 76 of them so we can use 299 00:22:55,540 --> 00:23:02,220 and know or Zotero to use our proxy to pull as many of those as it can as you may have 300 00:23:02,220 --> 00:23:06,680 experienced it's not always super accurate at that so we usually figure it gets about 301 00:23:06,680 --> 00:23:12,380 50 percent of that and then there's like another 50 percent that you have to go pull by hand 302 00:23:12,380 --> 00:23:19,020 or in order via document delivery interlibrary loan if there's a lot of papers what will 303 00:23:19,020 --> 00:23:24,140 we all usually do is we'll sort of divvy it up amongst the team and so everybody goes 304 00:23:24,140 --> 00:23:28,460 and pulls 20 papers or something like that so that it not it doesn't land all on one 305 00:23:28,460 --> 00:23:34,300 poor person so that's one thing that we do also the library provides a free document 306 00:23:34,300 --> 00:23:40,180 delivery so if we can get the paper for you from somewhere else besides Duke that we will 307 00:23:40,180 --> 00:23:45,720 and and you should do that when you're doing a systematic review if it's not at Duke you 308 00:23:45,720 --> 00:23:52,580 are sort of obligated to go get it if you possibly can at another institution leaving 309 00:23:52,580 --> 00:23:58,780 it out just because we don't have it in full text is is not a legitimate thing and I actually 310 00:23:58,780 --> 00:24:05,020 do I'm a reviewer for a lot of journals I'm right I'm the librarian methodology expert 311 00:24:05,020 --> 00:24:12,100 and if I see that in a manuscript it's a very quick reject now speaking of leaving out papers 312 00:24:12,100 --> 00:24:18,900 one thing that I see that that I struggle with is the language the language thing so 313 00:24:18,900 --> 00:24:24,140 if something is not in English how do you decide what to do with that what is the best 314 00:24:24,140 --> 00:24:32,580 practice versus kind of like what's most practical okay so I am a huge fan of including as many 315 00:24:32,580 --> 00:24:40,420 people as you can on your team so and a multi-disciplinary multi-national team is fantastic because then 316 00:24:40,420 --> 00:24:47,980 you have native speakers that can then go and translate those papers we always keep 317 00:24:47,980 --> 00:24:55,200 all languages in the search and this is so that you're not doing you know a western bias 318 00:24:55,200 --> 00:25:00,220 in your search for information great research is done in China great research is done in 319 00:25:00,220 --> 00:25:05,080 Africa great research is done in Germany you can't just because they don't speak English 320 00:25:05,080 --> 00:25:10,140 is a primary language that they can write a scientific paper in doesn't mean it isn't 321 00:25:10,140 --> 00:25:16,180 good science so it's a very rude of us to be like yeah they wrote it in German and it's 322 00:25:16,180 --> 00:25:20,580 probably not worth looking at right let's so we try not to have that sort of English 323 00:25:20,580 --> 00:25:26,500 language bias so we included in the search so that you can at least have a look at it 324 00:25:26,500 --> 00:25:31,660 what most people do is they don't end up unless they have someone on their team that speaks 325 00:25:31,660 --> 00:25:37,340 the language they don't end up using those papers because it's very expensive to have 326 00:25:37,340 --> 00:25:42,900 them translated several thousand dollars a paper I have a friend who does this so I've 327 00:25:42,900 --> 00:25:48,340 got the skinny from her and she's like yes very expensive so we don't usually do that 328 00:25:48,340 --> 00:25:52,900 but you've shown you're like look there's this paper in Polish and there's this paper 329 00:25:52,900 --> 00:25:58,780 in Afrikaans and they really seem because the title and abstract are in English they 330 00:25:58,780 --> 00:26:02,940 really seem to address our question but we didn't have the resources now if you know 331 00:26:02,940 --> 00:26:06,420 you're going to be doing it maybe scoop a little bit of money in your grant off to do 332 00:26:06,420 --> 00:26:11,860 translations but if this is a surprise SR and you didn't have funding for it what you 333 00:26:11,860 --> 00:26:17,340 can say is here are the papers that we really think would be useful and if someone in your 334 00:26:17,340 --> 00:26:22,700 audience wants to spend the money to translate they can or if they speak the language they 335 00:26:22,700 --> 00:26:28,340 can go have a look at it but you're that you're addressing that it's there and not just ignoring 336 00:26:28,340 --> 00:26:32,820 it I usually encourage people to put in like a little table that said these are some papers 337 00:26:32,820 --> 00:26:38,620 that were interesting and we couldn't translate them but look we found them and I think that 338 00:26:38,620 --> 00:26:42,900 just makes the team look great because you're trying really hard to minimize the bias in 339 00:26:42,900 --> 00:26:44,700 your review. 340 00:26:44,700 --> 00:26:48,340 That's awesome thank you thank you for that perspective for giving us the ideal and the 341 00:26:48,340 --> 00:26:54,380 practical now the I think last question I want to ask is about authorship okay your 342 00:26:54,380 --> 00:26:59,380 librarians done all this great work and then you go to write the paper how should you be 343 00:26:59,380 --> 00:27:03,700 thinking about including your librarian as a co-author with you in the process? 344 00:27:03,700 --> 00:27:12,300 I love that you asked this bless you so we're right that if you think about and I guarantee 345 00:27:12,300 --> 00:27:16,540 that if anyone works with a librarian you're going to definitely want to give them authorship 346 00:27:16,540 --> 00:27:21,400 because we do quite a bit of work we're not just you know it doesn't take us five minutes 347 00:27:21,400 --> 00:27:26,900 to run the search and then you never see us again we help you with all your methodology 348 00:27:26,900 --> 00:27:31,940 we help you make decisions about what risk of bias tool you might want to use right all 349 00:27:31,940 --> 00:27:37,100 those things plus we will write up your methodology for you for the bits that we do so we can 350 00:27:37,100 --> 00:27:44,180 say this is how we conducted our search we provide you reproducible searches that are 351 00:27:44,180 --> 00:27:49,300 transparent that can go in your supplementary material I offer a lot of advice for a lot 352 00:27:49,300 --> 00:27:54,500 of other pieces of the review I offer a lot of advice on data extraction here's what I 353 00:27:54,500 --> 00:28:00,940 would do here's best practices so we usually just say you know we're a team member we don't 354 00:28:00,940 --> 00:28:06,780 have to be in on all the nitty-gritty things because I don't know anything about hematology 355 00:28:06,780 --> 00:28:12,300 or whatever your particular topic is you don't want me screening you don't want me data collecting 356 00:28:12,300 --> 00:28:16,460 because I don't know what I'm doing but I will be there to help you with the entire 357 00:28:16,460 --> 00:28:23,220 like process of doing a review so I think that work I think genuinely a lot of times 358 00:28:23,220 --> 00:28:29,140 are even part of conceptualizing your question so I do think that that helps us deserve being 359 00:28:29,140 --> 00:28:35,260 part of the author team and most of my teams have said oh absolutely you we feel like you've 360 00:28:35,260 --> 00:28:40,380 deserved it I sometimes every now and again I'll have one that was just super super easy 361 00:28:40,380 --> 00:28:43,740 and they didn't need me and I'm like you don't have to include me so if I feel like I didn't 362 00:28:43,740 --> 00:28:48,620 do the work come say have at it you don't need me on this but I feel like it also makes 363 00:28:48,620 --> 00:28:53,660 your review look really good look we used a professional librarian so there's been someone 364 00:28:53,660 --> 00:28:58,380 here sort of keeping an eye and then one of the things that I do is I go through the manuscript 365 00:28:58,380 --> 00:29:04,180 at the very end because I'm also a reviewer and I say this is what I look at when I'm 366 00:29:04,180 --> 00:29:09,420 asked to look at a review what things did you hit what things are missing and so I feel 367 00:29:09,420 --> 00:29:15,340 like just that helps us sort of get over the finish line of publication absolutely absolutely 368 00:29:15,340 --> 00:29:22,980 thank you thank you so I'm thinking my final question has to do with tools what are some 369 00:29:22,980 --> 00:29:29,540 favorite tools software tools those kinds of tools gosh we have some really cool ones 370 00:29:29,540 --> 00:29:37,500 so at Duke we subscribe to software called covenants which is a systematic review screening 371 00:29:37,500 --> 00:29:42,660 software but also helps with the data extraction it doesn't do it for you we have not gotten 372 00:29:42,660 --> 00:29:47,540 there yet people always ask me that they're like does it do it for you and I'm like not 373 00:29:47,540 --> 00:29:52,780 yet not yet but what people are working on it they're trying but it does really help 374 00:29:52,780 --> 00:30:00,220 with the sort of data management of your systematic review so helps you your yes and no's tagging 375 00:30:00,220 --> 00:30:06,220 things deciding on your criteria all that stuff and it provides a really pretty prisma 376 00:30:06,220 --> 00:30:12,140 flow diagram for you at the end and all the good stuff plus forms for keeping all your 377 00:30:12,140 --> 00:30:17,380 collecting all your data and keeping it all in one ecosystem so we love that software 378 00:30:17,380 --> 00:30:23,180 there is other software out there we just happen to have this when we find this to be 379 00:30:23,180 --> 00:30:30,780 the most user friendly other tools I really like to use are there is this is super nerdy 380 00:30:30,780 --> 00:30:37,420 but there is a software out there called the polyglot translator which allows us to take 381 00:30:37,420 --> 00:30:43,100 the initial search that we do and it will translate it for you it doesn't do a spectacular 382 00:30:43,100 --> 00:30:49,540 job there's things it doesn't do for you but it does actually save some time with syntax 383 00:30:49,540 --> 00:30:52,620 so I teach some of the students about this and I say here's the things you're going 384 00:30:52,620 --> 00:30:56,620 to have to look out for but if you don't want to work with a library and this will sort 385 00:30:56,620 --> 00:31:03,220 of get you halfway there there is a super cool tool out there called citation chaser 386 00:31:03,220 --> 00:31:08,180 so one of the things that you're expected to do is when you get your final set of papers 387 00:31:08,180 --> 00:31:13,460 so let's say we have 10 papers at the end you're expected to look through the references 388 00:31:13,460 --> 00:31:18,540 of that we used to be called hand searching but what they now call it I've heard a lot 389 00:31:18,540 --> 00:31:23,100 of different things I call it forward and backward citation searching so you look through 390 00:31:23,100 --> 00:31:27,780 the papers just to sort of make sure that there isn't anything living in the references 391 00:31:27,780 --> 00:31:33,140 that might fit your topic but then there's also papers out there since let's say there's 392 00:31:33,140 --> 00:31:38,820 a paper that was done in 2004 and it's been cited by 15 other papers you'll want to make 393 00:31:38,820 --> 00:31:43,260 sure that you've looked at those 15 papers to see if they have if they answer your question 394 00:31:43,260 --> 00:31:47,980 so there's a software out there there's a couple of them the one I like is called citation 395 00:31:47,980 --> 00:31:52,700 chaser and you can actually just put in the doi of the papers and it will go get all the 396 00:31:52,700 --> 00:31:58,180 references and all the cited papers and and then you can just turn around and drop it 397 00:31:58,180 --> 00:32:03,460 into that software that covenant software and it keeps track of everything I just it's 398 00:32:03,460 --> 00:32:08,940 really really nice and it's super easy to use so yes those are some faves there's some 399 00:32:08,940 --> 00:32:13,060 other ones I like to use those are the ones that I think are probably used the most often 400 00:32:13,060 --> 00:32:17,380 that's awesome citation chaser is one I had yeah so that's that's pretty awesome thank 401 00:32:17,380 --> 00:32:22,740 you next time when you hit that spot we will we will touch it we will touch it I'll show 402 00:32:22,740 --> 00:32:27,420 you how now for anyone who's as they're coming to the end of the podcast I'm just wondering 403 00:32:27,420 --> 00:32:33,580 what major thing about about systematic reviews haven't we talked about that you want to highlight 404 00:32:33,580 --> 00:32:38,900 or maybe we've talked about it you just want to emphasize it we haven't really talked about 405 00:32:38,900 --> 00:32:44,980 the process of doing a risk of bias or quality assessment with papers that we mentioned it 406 00:32:44,980 --> 00:32:49,300 super briefly but that's one of the defining characteristics of doing a systematic review 407 00:32:49,300 --> 00:32:56,220 and I find that a lot of teams don't realize that that has to be done you can choose to 408 00:32:56,220 --> 00:33:02,140 do a review and not do the quality assessment risk of bias but you may not call it a systematic 409 00:33:02,140 --> 00:33:07,820 review at that point so you can call it a literature review a narrative review a state 410 00:33:07,820 --> 00:33:12,220 of the art something but you may not call it a systematic review without doing that 411 00:33:12,220 --> 00:33:18,340 risk of bias portion and what that is is just looking at the paper and and making sure you 412 00:33:18,340 --> 00:33:23,580 can only tell what they reported but there it's relatively easy there are a checklist 413 00:33:23,580 --> 00:33:28,780 usually that says you know did they ask a good question did they I'm going to use randomized 414 00:33:28,780 --> 00:33:35,260 control trials as an example did they randomize was the allocation blinded you know how did 415 00:33:35,260 --> 00:33:39,820 they crunch the data was you know all those kinds of things just so you can look at how 416 00:33:39,820 --> 00:33:47,780 well conducted this is really just how well reported a study was and so that way when 417 00:33:47,780 --> 00:33:53,500 you're summarizing everything at the end you can say and you have to include I've seen 418 00:33:53,500 --> 00:33:58,740 this done but people will throw out the poor scores and I'm like no no no no no because 419 00:33:58,740 --> 00:34:03,540 you don't do that either it's a little bit like getting rid of patients that don't do 420 00:34:03,540 --> 00:34:11,020 well in your study that this is no no bueno no you still want to report on them because 421 00:34:11,020 --> 00:34:15,380 someone might say well I thought that Smith article was great you're like no actually 422 00:34:15,380 --> 00:34:20,860 we looked at the Smith article and there's a really some weaknesses some really a high 423 00:34:20,860 --> 00:34:27,580 chance of bias creeping into the paper because of X Y or Z so that portion is actually really 424 00:34:27,580 --> 00:34:34,460 really important and then you can say look the quality of the research done on this topic 425 00:34:34,460 --> 00:34:39,660 is really quite good or it's a middle in or you know nobody's doing really high quality 426 00:34:39,660 --> 00:34:45,540 research on this yet so it's just another portion of a review that makes it a systematic 427 00:34:45,540 --> 00:34:49,780 review that makes it super valuable if you do a scoping review you do not have to do 428 00:34:49,780 --> 00:34:55,660 it but that whether or not is a scoping or a systematic review really depends on your 429 00:34:55,660 --> 00:34:59,780 question and a librarian can help you sort that out. 430 00:34:59,780 --> 00:35:04,020 That is super awesome I've learned new things today Leila thank you. 431 00:35:04,020 --> 00:35:05,940 You're welcome. 432 00:35:05,940 --> 00:35:10,020 Well yeah that's been a it's been a great great discussion about systematic reviews 433 00:35:10,020 --> 00:35:16,380 I want to thank you for coming on and sharing your expertise I wonder why is there a caucus 434 00:35:16,380 --> 00:35:23,100 for systematic reviews what makes it so so so impressive because you told me it's a it's 435 00:35:23,100 --> 00:35:28,500 the second largest group relative to other committees or other groups that you have within 436 00:35:28,500 --> 00:35:30,180 the library association. 437 00:35:30,180 --> 00:35:36,140 Yeah we have almost 700 members I think the reason one systematic reviews are super fun 438 00:35:36,140 --> 00:35:42,300 to do this is also a skill that librarians are really reviews as you probably know just 439 00:35:42,300 --> 00:35:48,500 getting more and more popular and medical professions are showing up at a lot of universities 440 00:35:48,500 --> 00:35:54,800 and also at you know pharmaceutical companies and right there's lots of places that people 441 00:35:54,800 --> 00:36:01,860 are doing literature review type research and so and librarians are really important 442 00:36:01,860 --> 00:36:07,820 to that process and so more and more librarians are being asked to participate and so we want 443 00:36:07,820 --> 00:36:14,300 to be good at this for our clinicians for our researchers and so there's a lot of sharing 444 00:36:14,300 --> 00:36:20,460 of knowledge we are doing a whole one of our librarians here I invited her to come speak 445 00:36:20,460 --> 00:36:25,140 about including gray literature and their systematic reviews and like how to find that 446 00:36:25,140 --> 00:36:34,740 so it's just a way for us to a little bit of shop talk and a little bit of some commiseration 447 00:36:34,740 --> 00:36:39,220 and also this is what's happening on the sort of bleeding edge this is what people are doing 448 00:36:39,220 --> 00:36:46,260 like is AI there yet can we use AI yet to do a systematic reason not yet I've had a 449 00:36:46,260 --> 00:36:51,060 couple teams asked me recently and I like not yet it's getting there but we're a couple 450 00:36:51,060 --> 00:36:56,400 years out so those kinds of things now that's really awesome I think systematic reviews 451 00:36:56,400 --> 00:37:02,220 are very valuable and so having the right people in the room to do them especially your 452 00:37:02,220 --> 00:37:06,700 dedicated amazing librarian is important so I want to thank you for coming on the show 453 00:37:06,700 --> 00:37:11,940 to talk about it today thank you so much this is really fun I will come back and talk as 454 00:37:11,940 --> 00:37:19,100 long as you want me to I will invite you again okay all right everyone you've heard it from 455 00:37:19,100 --> 00:37:24,660 Layla Ledbetter talking to you about how to do systematic reviews in the best way possible 456 00:37:24,660 --> 00:37:28,720 Layla if people have questions and they just want to connect with you how can they reach 457 00:37:28,720 --> 00:37:35,300 you you may go to the Duke University Medical Center Libraries website and you may find 458 00:37:35,300 --> 00:37:40,780 me there in the staff list we also have a guide on systematic reviews it's pretty popular 459 00:37:40,780 --> 00:37:46,060 if you google Duke systematic reviews you can reach us that way and we are happy to 460 00:37:46,060 --> 00:37:51,220 answer any questions that you have about the process and direct you to resources on how 461 00:37:51,220 --> 00:37:56,580 to learn how to do that there's a Coursera course done by Johns Hopkins that's fantastic 462 00:37:56,580 --> 00:38:01,700 so there's lots of places we can direct you if you'd like to learn more sounds good well 463 00:38:01,700 --> 00:38:07,140 thank you so much Layla you're welcome bye all right everyone have a great time we'll 464 00:38:07,140 --> 00:38:17,180 see you again on the next episode 465 00:38:17,180 --> 00:38:22,540 thanks for listening to this episode of the clinician researcher podcast where academic 466 00:38:22,540 --> 00:38:28,260 clinicians learn the skills to build their own research program whether or not they have 467 00:38:28,260 --> 00:38:34,020 a mentor if you found the information in this episode to be helpful don't keep it all to 468 00:38:34,020 --> 00:38:41,380 yourself someone else needs to hear it so take a minute right now and share it as you 469 00:38:41,380 --> 00:38:46,780 share this episode you become part of our mission to help launch a new generation of 470 00:38:46,780 --> 00:38:52,820 clinician researchers who make transformative discoveries that change the way we do health 471 00:38:52,820 --> 00:39:17,620 well