Cochrane Canada: Risk of bias assessment of RCTs in Cochrane Reviews with Lucy Turner

Cochrane Canada: Risk of bias assessment of RCTs in Cochrane Reviews with Lucy Turner

Lucy Turner addresses the issue of risk of bias in Cochrane Reviews. This session was recorded on Nov 20,2012.

like to send a quick thank you to the people who provide funding for the Canadian Cochrane Center which allows us to conduct these webinars II like to thank the Canadian Institutes of Health Research and the six Institute's that are listed there without their support these webinars wouldn't be happening like to send a special thank you to PAHO and w-h-o that is the Pan American Health Organization which is a regional office of the World Health Organization they're providing us with the software so we can provide these webinars free of charge can I ask everybody to send a quick round of applause through to paho using your motor cones on your screen a round of applause through to Tahoe thank you very much everybody so today's session doing a risk of bias assessment of our CTS randomized control trials in Cochrane reviews in today's session is going to be led a recent Turner Ruth who's one of our expert presenters is based in Ottawa at the Ottawa method centre part of the knowledge synthesis team for statistical work ranges from analysis of survey data to meta-regression accumulative meta-analysis if you have any questions about statistics or the risk of bias receive certainly the person to go to she's a research coordinator said Cochrane bias methods group so Lucy thank you so much for presenting today and I'm going to turn it over to you to go ahead hi Erin thank you very much that introduction as an exists my name is Luz Eterna my background is in statistics I've been coordinating the Cochrane blast message group for about three years now working with David my own knowledge since this group here in Ottawa in Canada and please if you're having any difficulty hearing me please let me know and also I do have a tendency to speak a little quickly so if I'm going too fast this particularly for those whose language first language isn't English please do let me know and any feedback is welcome so please don't hesitate to mention that using the Cockrum risk of bias tool is this session is going to be a practical example and will give you a bit of a session overview in just in minutes as you'll have access to these slides and I thought it was important and have an introductory slide with additional materials that would be helpful to you perhaps afterwards and it's definitely not possible to learn how to conduct risk of bias assessment within an hour but hopefully we can give a very good overview and these additional resources are here and hopefully we'll be able to help you in the future prior to doing your risk of bias assessment and I'm just wondering if we could have an impromptu poll right now just to see how many people have used the risk of bias tool before thanks Lucy so I'm just setting up a poll right now if I could ask you to just wait one moment there we go and if you could use the polling function yes or no send through a yes or no if you have used the risk of bias tool and will give just a moment for those responses to come through and then I'll publish them to the white board Lucy thank you oh I'm sorry that was a little off-the-cuff thank you and so just in terms of the Cochrane handbook it's readily available online there are some changes that will be coming through in the next little while and so it's a good idea to always check out the the handbook prior to conducting your risk of bias assessments as there will be some modifications some minor modifications now and some larger scale multiplications that will be coming to in the next couple of years okay so it's interesting to see the results of the poll quite a few people those people said that people haven't used the risk of blast all before and it's quite evenly split so so that's good of those 18 people who didn't respond and we all err on the side of of other no sorry if the foot pole isn't isn't likely there are also training slides available and ride the training Western groups thanks to them for organizing those and setting those up their comprehensive slides that goes through how to assess risk of fast and they also go through reporting biases reflect those are very helpful resources and please do always feel free to get in touch either by going to the PMT website which has a number of online cases and references which may be of interest to you and you can always contact us directly and my email address is there so just quickly and my own acknowledgement to the vise message group conveners professor Oakland doctors patron and Robison and dr. David Nora who have given input to these slides and continued to do a lot of work for our group also back to my aunt Constance who who developed a suite of the standardized training type and we also share the same fundus for those who found the webinar to affect the CIHR in terms of a session overview as I mentioned it's very challenging to teach risk of fire within an hour and sometimes we have two very sessions and we can get through and into the nitty-gritty a bit more thoroughly so what we're trying to say is to work through an example which hopefully you will have in front of you a fairly short and concise example and hopefully it will give a good introduction to conducting a risk of fire assessments we will not cover anything on reporting biases today and will not cover anything on assessments for non randomized control trials today you may be interested to know that the BMG is currently working on sort of a three-pronged process for updating the way that we assess risk for fire in Cochrane reviews the first will be risk of buyers 2.0 which will be some sort of format change and not sure what happens okay which will be some more to the format change and then there's also a group of people at the University of Liverpool and working in conjunction with the BMG assessing reporting bias and particularly selective reporting so some work going on there and there is a huge initiative that's going on around which involves over 50 senior methodologists who are developing a suite of tools in extension to this current version of the risk of 5 tool for assessment as a non randomized study we also learned other in great detail other issues and with regard to bias assessment today but they will be detailed in the shoes be updated version of the habit so inside the steps of a Cochrane review where are we with this process so you've gone through your your screening you identified your studies and you've collected your data and now you're assessing your studies for potential risk of fires these are fairly interchangeable with data extraction you can do it simultaneously you can ask to people on your team to do data extraction to promote entity risk of bias assessments with you suggest that whisper bass is conducted independently and in duplicates hopefully with a methodologist and a clinician in combination as clinicians sometimes are able to give input that a methodology stream of C and and vice-versa so coming to consensus there would be helpful in terms of the process given that the process is reasonably subjective and we do ask that risk of bias accession isn't an afterthought and that it's integral to the process of your Cochrane and in the non Cochrane review and then it's not kind of done at the end in a hurry because it's something you have to do it is actually obviously quite unfortunate and can influence the findings and the overall findings of you are really quite fun actually so it's something that should be taken cor seriously so before we go any further and start delving into the practical example it's really important that we think about what is bias and that we're all on the same page in terms of and what more sort of bias is or is not so bad for the systematic error or deviation from the truth so this is saying where people say garbage in and garbage out if a number of your studies are giving inaccurate estimates of effect or inaccurate results this can obviously we need for misleading reviews or results for your review and the main question you should be asking yourself when you're assessing risk of glasses can I believe the results that I have it's important to note that bias is not the same as imprecision so it's not random errors you too startling it's not the same as quality so you shouldn't be asking the question was this reserve study conducted at high quality it may be that the investigators did the best they could with the constraint that they have for example of surgical trials may be of the highest quality that it can be but you cannot blind it so there is still potential for risk of bias with those types of studies and it's not an issue of reporting so this method may or may not have been done that this is poor reporting we're really it at a loss really in terms of risk of fires assessment and nothing less about quality scales and checklist and Agnes teashop published law seeing where in annals a paper looking at how quality amount of my control trials is assessed and identified many different ways over twenty six different ways of assessing risk of bias or quality in primary study within Cochrane reviews we suggest that qualities that girls should not be used PC uni has some empirical evidence which suggests that skills as a result of using scales and aggregate scores may be misleading so so a convalescent within that framework although that makes it slightly more challenging to assess if you don't have a summary score some summary value and makes it a bit challenging to assess which shows at higher risk for sparse compared to others and but this is a complex process and obviously timeless sort needs to go inserts and summarizing is them it's not beneficial for us so what about the Cochran with the fire assessment there are seven evidence-based domains I think that's a really important point to to reiterate is that all of the domains in the Cochran risk of battle the exception to attrition by currently are a given so if there are methods in your logical studies which results will have resulted in the potential for presence of a systematic error or deviation in the truth subject to attack statistic is blinding or second-generation allocation systems two examples so both of the review authors judgments so you go through each of these seven domains and then you judge at low high or unclear risk as we're not scaling to the categorize and then you give the supports of the judgment so these evidence look quotes from the paper or some sort of explanation that you've come up with and then you in put these into web man or if you're not doing a Cochrane review and some sort of table and appendices to add to your publication these remains to address random speakers generation allocation concealments blinding of positive stuff systems and personnel and this will familiar with the tool you may be aware that this is recently been split because we're trying to categorize by five type systems business performance edge detection biased so for participants and personnel and blinding of outcome assessments instantly outcomes a belches attrition by selective reporting and other bias II and I underlying selected reporting and other biases and from platform these briefly today that we were going to them in detail with regard to the example and they can be quite challenging to assess and please consult the handbooks and there is far more explicit guidance than I can give today with regard to selective reporting and other biases and as I did mention we are doing work on selective reporting and that may make change in time column the way with the reassess selective reporting does anybody have any questions at this time Aaron perhaps if if there is one to question you if you can hear me now you please give a smiley face great good so moving on I don't think going to be a funny question I apologize if and if I missed any questions there we seem to have a bit of a technical problem but that's okay please do always feel free to email if you have any questions or specific questions well and your cream cream dip conducting systematic review so now let's move on to the example and hopefully you have in front of you and the randomized control trial that we that we forwarded to everyone it's quite a short RCT that's why we picked it for this session and it's looking at the effects of Tai Chi exercise on a number of different outcomes for older women with osteoarthritis it's an RTP and 72 patients around mind to to arm the first arm getting a 12 week 12 weeks western session of the Tai Chi and the other not and they fixed it before and after and p.m. post so let's go through by domain so you have two slides for reference later when you go back to the slides if you need to and we'll go through the main by domain and for each of the domains we'll have a pole and then for each of the domains we'll have any questions and with regards to this our cities so what is random sequence generations it occurs at the start of the trial before the allocation of pictures and this avoids selection bias so turning surrounding order of assigning people into the intervention and control groups and it avoids systematic differences between group how do we assess this and for those low and high risk and I should probably say now that we're trying to empower through in 2009 Lisa heartening and at the University of Alberta published this study and more recently there's been a recent paper this published this year and suggest that many of the tools that we use for assessing bias including the risk of life and there was also Newcastle author etc are not hugely reliable perhaps you can start like that and their papers were found to be quite low but especially across certain with the fires tool domain we don't find this surprising by any means especially because it's such a subjective process and it's a very localized process in terms of what you're doing for your systematic review and how you aim to reach consensus however we will be working on improving the guidance and if you do have any questions at any stage whenever please do ask if anything is a tool unclear in the handbook or underneath your guidance available please do let us know and we will try to improve that but one suggestion that we are making at this time is that instead of rating unclear if it's all possible to pick of them they're to pick a judgement so high or low we do find that a lot of people just kind of go for unclear but it's hopefully we can improve that information allowability with low and high judgments so low reps or random sequence generation would be random number table computer rather number generator if they use a method of minimization or these satify the block randomization and also low tech and options are available at coin tossing or shuffling cards some how to generate a sequence for randomization high risk of bias is caused by random so they suffer for basis visit or ID levels or something to that effect is not insufficient randomization and non random practice is also obviously not as my secret generation allocation concealments it's again at the beginning of the child and avoid selection bias but then is when people are recruited to study and ensuring that no one knows which groups that they've been allocated to you two alone risk of fire it has to be unpredictable so central allocation or sequentially numbers sealed a paper envelope in high risk of bias means that someone on the staff or otherwise to predict the random sequence so for example if you generate this beautiful essentially randomized list of ones and zeros for allocations to arms and then somebody puts it up on the walls but which patients coming in next obviously that horrifies allocating concealment so looking at the example and in I just wanted to highlight the text in the paper so this is under the methods section it's the first paragraph in the methods section and it details that randomize process was conducted by a hospital Coordination Center using a randomized table with no involvement of the research team so this is now time for our poll we'll do to poll the first which will be for sequence generation and the next one for allocations in film and please so let's do the first one first the sequence generation if you think it's high risk of bias please click a low risk of bias B see fun clear and D if you're a little angel thank you so the locust applies when you're getting up there now great great okay and as I mentioned earlier Felisa study does suggest that there is some there are some variability and it is reasonably subjective here and you will be doing it in to to cope with somebody else so you know you can come to consensus about these responses and there isn't it's not necessarily a right or wrong answer so talk about allocation concealments and again high risk low risk for unclear or a bit confused you Oh you okay so it's quite a few people saying low-risk a couple of people saying unclear a couple of Hitler's a little less poppy from collaborating okay so selection bias if you're looking at sequence generation and they've used a costal coordinating Center so if you go back and perhaps look at the Cochrane handbook there's a table in there and for low risk of fire and coordination by a centralized means of randomization was deemed to be low risk for fire allocation concealment as there is no involvement of the research team and they can force abyss sometimes it's it's often not reported and this is also very specify so they've implied here there's no way that people will be able to know the sequence that's been generated so the allocation has been concealed to the research team so that is why we've decided that felonious proper allocation concealment and seeking generation is also very viscous by does anybody have any questions nope okay and if you suddenly do decide of of course at any time you do have questions about selection bar please do get in touch so moving on to the next to make which is blinding of participants in personnel so this is performance five so performance bias is different treatment of in terms of intervention group it's different participant expectations and it can lead to changes in your actual outcomes blinding can be really challenging to assess and sometimes it's not reported and the language is really vague but what just thing or blinding mean or just double those blinded randomized receive your control trial and people often watch put those in their titles thinking that it will get them published but it doesn't actually they don't actually go on to explain what that actually means that comes back to our issues with reporting so it's important when we're assessing blinding to try and decipher whilst is meant by single or double blinding who is actually blinded it's also important and hardly ever reported and to affect whether or not you think there's a potential for blinding to has been broken so that's something else to think about and it's also important to consider that even if it's not feasible to this intervention so the example again of the surgical trials there's still a potential for risk of fire but just because they they can't blind the surgical intervention doesn't mean that this potential for bias doesn't exist so it could performance fire for participants and personnel load of the biases that their world's blinding that blinding occurred and it is unlikely that binding could have been broken and also it's very important to consider subjective and objective outcomes for both blinding domains so if blinding there wasn't any blinding or the blinding of incomplete or broken in some way if you have an objective outcome for example mortality it's very unlikely that this will influence the results or which is the result in a systematic deviation from the truth with regards to fire so high risk for fires is there's no blinding the blinding is incomplete or it's been broken and it's a subjective outcome or in some ways the outcome is likely to be influenced because of the lack of blinding so as I just touched upon for both performance and detection by so looking for outcome Assessors and for participants and personnel those two blinding domains it's important that you assessed by outcome so it's important that you consider it your outcomes or potentially a class of outcomes for example if your outcomes are all treated in the same manner so as we dis example for example we can we can look at physical containers there's a class of outcomes and think about those separately as some may be subjective as there may be more objective outcomes and the influence of lack of blinding will differ based on your outcomes oh and does this bottom point does notice that in Redman you are able to design your table for more outcomes so if you would like to look at all your primary outcomes or classes of outcomes you can modify the risk of bias tables within Redman to accommodate that so this is poll three and what do you think about performance by overall please vote hi is a low risk the bias is b.c.e is unclear with the bar and D if your your Civic compute you just wait a couple of seconds okay so most people think it's at high risk of ice some at lower fire some people are unclear and a couple of people are still a little bit confused so we'll try and reiterate here with the whisky example so there's no caption for this domain in particular so if you think about who has been blinded there's no information in the RTT but if we think about the intervention the patient's know whether or not like it's like a vertical trial the patient's know whether or not they've been doing Tai Chi for 12 weeks so it's very difficult to blind the patients and this is a change in their exercise behavior and also the care providers through administering the exercise program and they are also not blinded and again just to reiterate it's important that we think about our outcomes here for the self-reported subjective outcome sort of a map score and the physical performance which is slightly more objective so we as a BMG convenient or making to do this together and talk about this we came back please unclear risk of fire simply suppose we found that it wasn't recorded and they haven't given us any information about potential contamination so people that have been in the placebo group not taking any exercise and that have thought slightly more and moved on and they started taking up exercise themselves although there is a statement in the method center section which stipulates that they were telephone and to make sure that they weren't doing any exercise but this hasn't been reported in the results so we don't actually know whether or not people have taken up exercise so we thought this was this was somewhat unclear in terms of Co interventions and contamination but if you if you thought it was localized or high risk of fire for whatever reasons as long as you report your judgment and your rationale this is this is all we can do it's a very subjective process and my opinion today might be different to my opinion tomorrow is dependent on what side of bed by a couple and and that made it different from your opinion as well so that's why we again why we do it in pairs and try to come together this and there is no right or wrong answers does anybody have any questions at this stage David hi Lucy sorry everybody for the difficulty with the connection earlier there are a couple of questions that have come through the chat room we have a question from Robin asking what if someone gets a different later treatment because of lack of blinding and the later over FEX affects the overall survival what if someone gets a different later treatment because of lack of blinding and the later affects the overall survival Lucy could answer that I think so I hope I've understood this correctly so but I'm sorry and I'm not too sure why someone would receive a later treatment due to lack of blinding so um but with regards to performance or more smoke detection life if there's a difference between in the way that the intervention was administered or some sort of difference between groups it will provide unbalanced I think you should include that in other potential risk of fire section I don't think it would necessarily fit and in into performance but per se but I think need to definitely include that if somebody if there's a difference between the groups so for some reason and there is a delay in the intervention and in one of the groups compared to the other groups or even within a subset of the population out including other other risk applies I hope that it's from some level thanks very much Lizzie and Robin certainly if you had any further questions about that please do feel free to email Lucy and she can answer that outline there's another question online Lucy from David who has asked if blinding is not possible how do we express our judgement so for example if the intervention is a housing assistance it's impossible to blind do we still say high risk there's no not applicable option Lucy hi there is no there isn't enough accessible and option and if you believe the blinding may influence the outcome so there is a subjective outcome for example housing is provided by the states and how does that make the tenants feel for example some sort of staff report is subjective outcome and after they'd have moved in I would say yes and if you can't blind for that then that there would be a potential to influence the outcome and therefore I would I was being at high risk of bias for that study bias can still occur if the quality of the study was the best that it could be and given the context and the question in hands and if there's no potential way to blinds and the potential risk of bias can still exist Thank You Lucy there's our further question in the room the last one so far from black hole asking for dividing outcomes into subjective and objective are you referring to the schema offered by Leslie woods BMJ paper in 2008 is there additional guidance out there I love you I'm so glad that you could join us today thank you and thank you for being here yes does that is definitely and grateful me entitle evidences of with BMJ paper that was published in 2008 and in terms of additional and guidance there are a number of references that I should I look for my memories pouring out a KL from last year published one and and as John and Isabel have also done a lot of work on blinding and I will get the person of blinding references or most up-to-date ones for it for you and and send them send them to the group I hope I hope that's helpful everyone are there any other questions those are all the questions in the room right now thanks Lucy great thank you everybody and I do know that a couple of people are still a little bit unclear and in the interest of time and will cab will carry on at this stage and there will be more opportunity for questions as we go and if there's anything that's still baffling after the session please please do get involved get in touch so moving on to the next amaze is blinding of outcome assessments so participants and Personalis performance part and outcome assessment lining an outcome assessing is detection bias so it's the measurement of outcomes affected by knowledge of the intervention received and again it's very similar and issues with performance buyers with regard for wording of single and double blinding and it may be feasible and that there is detection bias even when blinding of participants and kept violence is not possible and and you should remember that parties and business and personnel may also be outcome assesses for that something to bear in mind for this RCT so louis provides very similar to means and to performance part is that blinding has occurred and it's unlikely to be a broken or that there's no blinding but they're very objective outcomes so your outcome measures are not likely to be influenced by lack of blinding social example mortality and you know if there's no blinding and the main major outcome for a third quartile is mortality then the influence of blinding may not have a greater effect and that can be deemed low risk bias high risk of fire is no blinding or gross and blinding and the measurement is likely to be influenced those self-reported or subjective our punch blinding of outcome assessments so this is what we have in our RCT in our phone our CC that we have and it's under this data collection procedures stop heading in the message center section so all subjects attended sports center of University Hospital for pre and post-tests measures of balance muscle strength and cardiovascular functioning by exercise physiologist using blind procedures it's time for two more poles the first pole and so that quite quick and lots digest in one go please feel free not to votive if you're uncomfortable so to use both on detection blasts and for this one we're going to break it up by at us so the wall map score is self-reported and fairly subjective do you think this is high low or unclear with the fire okay if it's low on use on us okay you you you okay a little bit more split very easily split actually between high/low and unclear so the women school and what about for the physical performance or more objectives goals what do you think for detective eyes you don't forget to think about who is the outcome Assessor and from whether or not they're objective or subjective outcomes can be Louis cobot okay a little bit more on the Louis – five sir – the objective outcome which is which is good to see and subjective come with a little bit more rare diversity across degrees there okay so if we think about who was the alchemist effort this is the patience for this honors it's self-reported they're assessing their outcomes and they weren't blinded it's also very subjective outcome but I would listed this high risk of life the physical performance these are a little bit more objective and the out confessor were the physiologist and they were blinded and it was a using a standardized measure set and I can see everybody sitting that computers Nikki Horan compile the blinded procedure was questionable they didn't give us any details about the blinding procedure and that these outcomes are more objective I would say they weren't specifically objective and so there's been debate within the BMD executive some of us would say unclear and some of us would say low and so of us a high risk of life so I think it depends on your standpoint and whether or not you think it would influence the physical performance results it was actually interesting that the clinicians in our group but for more high risk of fire and the methodologist in our group went for more low viscous part so that they there's no right or wrong answer to this and if you're ever in doubt please feel free to so have a chat with us and if you wear for low viscous ice that's perfectly acceptable having said that it's probably more beneficial to air on this on on the side of caution does anybody have any questions about outcome assessment you you no no they're excited people just typing in Lucy so we'll just give them a second to write in we've had one question come in asking what is the reference for the new Cochrane categories for assessing risk of bias high-low I've checked the book and it's not there yet a couple of other questions that have also come through asking in performing risk of bias assessment would you ever contact authors to ask for further details or just assess based on information reported if you process the 51 that's a very good question and I think that's completely up to you and your time frame your funding and particularly how many studies you have included in your systematically if you only have three or four it's perfectly feasible to get in touch with them if you you have a hundred of 200 or 400 I know that doesn't happen very often but if you do and then contacting authors may not be the optimal way to go but it is perfectly acceptable and we've had some internal conversations within our KS group recently about how many times it's acceptable to contact and author if you get non-response for them and we decided it was only twice and I think it also depends on how old the study is as well if you're including studies from the 80s or even the 90s and the likelihood that you'll be able to get in touch with the but authors they would have accurate information for you may not be and a hunch that reliable or even feasible but then otherwise I don't think yeah perfectly acceptable to to get in in touch with try lists and authors and ask them about their settings thanks Lucy another question that came through the room asked I think that outcome assessment is easier when we have a tool or scale do you think that's true you I'll just skip a moment for Lucy to come through KCCI unclipped my myself and yes I do believe it's easier and to have scale and I do also believe it's easier to have a family scale and there's a lot of discussion within the vmg at the moment around summarizing risk of bias and just to make a little more digestible both for authors and for clinicians in terms of interpretation and it is fairly messy practice but although it easier to have a summary score that's almost implying that there's equal weighting by the main so you're saying the sequence generation has a similar weighting in terms of potential effect on your results as performance bias or detection bias which may indeed not be true depending on the context of your review and lack of blinding may have a huge influence on your results in your systematic review and whereas you know selective reporting or another type of risk of bias would have a major impact on your systematic review and so the scaling gives giving equal weight to each domain is and a mirror thick and can be totally misleading in my opinion thanks Lucy there's one last question that's comes through the room from David asking when you're personally not sure and you're hesitating so for example between unclear and high-risk do you tend to be more conservative or do you tend to give them the benefit of the doubt that's a really good question and I think it depends and as I mentioned early on how I feel on the day if I'm in a bad mood I probably get high with the bus to be honest but hopefully the person I'm working with in duplicates in a better mood and I would be more conservative yes I would tend to go some more high risk of fire than Larry Oh survive I think there's been a tendency from across the board from bosses and not to mark high risk of fire and it's not displayed against also it but nothing to do with the quality or the conduct of the trial and it's not a personal attack by any means to label somebody's results potentially at high risk of bias for the results of your review within the context of the question that you're asking and so yes I would be more conservative and depending on the field for the paper and I wouldn't give the benefit of the doubt if I'm brutally honest thanks for the season there are a couple of more questions that comes that have come through I wonder if I if you'd like to continue at this point and we'll save Murray and ones questions to the end if that's okay Lucy what do you think yeah perfectly okay so I've just realized the time man we reaches get a move on thanks again and we'll take those questions later on so incomplete out some paper so this is attrition but so official can happen for many reasons so it's been complete outcome data for all participants is not available in your review so there's lots to follow up this is rules people have moved there's other reasons for missing data or people have been exclusive so some data that should be available is not included in the report and this all needs to attrition fire there's no empirical evidence for intuition bias at the moment but logically it makes sense if there's been some proportion of your trial participants that have dropped out for ever those reasons maybe there's differences between groups there's some sort of imbalance this could result in a systematic error in the results for this value and then within subsequently within your systematic review so things to consider for attrition bias how much data is missing for each of the groups why is it missing so the reasons for exclusion or withdrawal or lost for lot other missing data and given provided and is there an imbalance between the group two groups and how does the data analyze one of the biggest questions and problems that we have with assessing with the fire is how much is too much missing data and this year last month actually and Jelena Salim edge who's a very active biased method group member and in collaboration with with many others publish the brand own report so they're looking at fire characteristics there's a compilation of all the meta epidemiological work that is out there so the empirical evidence that drives while we assess life and what fires domains we assess and they were looking at attrition by so they took a cut point of 20% and there was no systematic error with regards to the virus results that with regard to the results of individual studies so the most simple rule in terms of how much is too much missing data and maybe it's not 20% at the moment we just don't know and it's very difficult to assess that but what you have to do is is the missing data that can meaningfully affect the results of your study so the overall proportion of missing data the event risk for the discussion of outcomes how much that would change and your event risk for example if you have rare events and there's there's a large number of dropouts what with that that mean how that intruder results and the plausible effect size and very similar and reasons related to study outcomes so and was it an adverse event and so refused or the reasons for refusal and for their certain demographic of population that's refusing to participate and the missing base or reasons were not balanced between the two groups so one group intervention group had a severe number of withdrawals due to adverse events and the control group didn't for example that could be a potential for attrition by and just gonna quickly skim over intention-to-treat analysis please reflect upon this while while doing your assessments and assessing incomplete beta it may be difficult to reach the conclusions I were mentioned so lose Louis provide no missing data and the reason for missing data is not related to the outcome there's missing data and there's an imbalance between the groups there's a balance between the pictures of even similar photos is the lowest of us the proportion of missing or plausible effect size is not enough to have a clinically relevant effect and the high-risk advice somewhat converse to those reasons so the reasons to attrition are related to the outcome and that there's an imbalance in the numbers for the reasons between arms the proportion of missing or plausible effect size is enough to have a clinically relevant effects as creative analysis was conducted with substantial departure from the allocation we can talk about that at a different stage and inappropriate use of implications with regards their analysis so we've also our randomized control trial example if we took 42 female participants were random on those dropout rates of 43 percent and 39 percent in the experimental and control groups respectively so what do you guys think about this in terms of what we're thinking about with regards to proportion of drop out there are also reasons given the drop out but they don't specify which groups and has the reasons for the drop so it's impossible for us to determine balance for the reasons of dropouts and his initiative table and where they've finally done their analysis on pre and post available data which is for 22 and 21 in the exercise of control arms given that 72 remanded lines so what do you guys think this is the final poll do you think it's a high profile for a low risk of fire for being seniors unclear with the fire and D I provide the fuel for still a potential and it is not easy to tell makes players to come in high risk of fire quite a few people you you you okay great many people are saying high risk of bias few unclear and also if you are low which is which is five weakest tandems that we thought it was high risk of bias purely on the percentage of dropout so although it was reasonably balanced between those arms in terms of the number of people who were drew from the study and we thought that it was at high risk of life for infancy for infancy outcome data just due to the sheer number of people who dropped out just to touch very quickly upon selective reporting and selective reporting can lead to reporting via feeds this is where there's a difference between what has been planned in detail reported on clinical trials of glarb or in the study protocol with what has been reported in the final publication this is incredibly challenging to assess as a risk of bias item then there is work going on around collective reporting to try and improve the guidance and help people with this but if we don't have the protocol or the registered trial available to us it's very difficult to determine what has been done and that's an issue from a reporting issue as opposed to means of assessing selective reporting for many of the judgments are unclear and although we try to encourage people to steer clear of unclear and judgments and with regard to selective reporting it's very common to have unclear supply with this judgment see so the basics main theory so just one quick spot I noticed that we're hitting one o'clock and I recognize that some people are going to need to lead the room just to let everybody know that we will be sending through a copy of the slides and we'll also follow up with an evaluation form by email if you do need to leave if you have another commitment please don't worry about missing from some of the content we will follow up with that by passing that along sorry Lucy notice the tool thank you and I'm very sorry that we've that we've gone over time and and I do hope that you have chance to refer to the to the PDF of the slides and if you do have any questions I don't think we haven't covered that you're not not able to carry on listening to Gigi and let me know thank you very much for those you have to to me there's also a section in the Cochrane handbook on other resources advice and it should be important to note that these do not include imprecision so small sample size that have a lot of potential risk of fire and inadequate dose or an unusual population for diversity or hex genetic but is not a potential risk for fast and other measures of quality so whether or not those ethics approval or have funding with obtains that's a big one and they're not other sources of bias at this stage and there are three potential issues but the dmg is investigating at the moment with regards to Charles which has been stopped early for benefits and the difference between single Center and multicenter trials and whether there's a difference in measures and effect and we're also looking at source of funding so those are outstanding issues which are not necessarily sources as well and should not be considered at this time in the risk of riot or as as it stands for needs of time this is an example of the risk of fire table in red map so you use the mains on on this side on the left hand side where it says by the heading underneath which you can modify these and change them as you wish and based on on the guidance of the boilers and then the author's judgment is where you include your high we'll know or your unclear and support from judgment which should be completed and now in line with the messiest and methodological expectations Cochrane interventions reviews which published and is available on the Cochrane websites it's mandatory to complete risk of five tables in this form for Cochrane reviews there are two figures and which are produced investment to give a nice overview of the risk of bias assessments these are our ways of summarizing without a scale without a summary school and so the green is no justified for very high risk advice and so it's a nice sort of graphical illustration and this one's very similar lazy is white this is where from on his negate is to include that assessment they haven't included that on there so this is this is what we can do in terms of summarizing some questions that we currently have within the BMG are should we be done by outcome so should we be looking at overall risk of our successes where outcome rather than by study and should we be waiting Lee for example and these are you kind of eats equally distributed for each of the studies so it may be interesting to break those based on sample 5 or perhaps there's another way to do that so that's one of the questions that we currently have at the moment there are a couple of slides here and for the four time I'll quickly go over these but not in too much detail so incorporating your findings into your review this is incredibly challenging and not many people do it and one of one of the big kind of marketing campaigns not really but one of the big drivers at the moment is to find encouraged people and to include a description or descriptive narrative description of risk of wire across domains in shorter summary as they can in their discussion section but this may be missed and it doesn't consider the impacts on your results so it may be possible for you to restrict your primary analysis to notice the bus and have a look and you can always conduct sensitivity analysis to see if risk of bias assessments influences your meta analyses a little bit harder for qualitative synthesis and you can always present a stratified analysis and you can also explore by subgroup analyses or methodological metal regressions and if you feel so inclined and particularly if you have a large number of included studies you can point what prioritize amazing your view as we said earlier for example in some studies blinding may have a large impact or you would perceive having a large impact on your results and it's important to try and think about the potential direction of the impact so perfect bias is flat bias due to lack of blinding taking my measure of effect towards or away from the null and this can take some sort some time so it's important that risk the fastest isn't a mandatory inconvenience death and you actually give time for thinking ok so I put these risk of bias assessments I've got my outcomes I've got my summaries across and this very large study is high risk for fires for selective selective reporting what does that mean how is that influenced my results in which direction I'm reaching an overall interpretation apologies the rush there at the ends and does anybody have any questions thank you very much Lucy there are a couple of questions that we had from the last question session and we had a question from Murray who asked if you contact an author they provide a response that alters your rating of bias do some help like this in the table or do just word accordingly and it did spray up to you I was encouraged and transparency and I would definitely encourage you to report that also have responded to the questions that you have and I would definitely update the rating accordingly and that you could also note that in the judgments or in the characteristics tables we carry English proficiency thank yourself a question from one who laughed when assessing the overall risk of bias of the meta-analysis do all individual trials weigh the same no matter the size or other characteristics yeah that's that's a great question and then so I did rush through that at the end they do currently take on all of the same waiting and they're not grouped by outcome so it's across studies and that's something that is important for authors to consider when there is nothing that they were surprised by outcome and also by trial weighting and I agree and I think we will be moving in that direction that obviously the impact within a rate of meta-analysis is that larger trials will generally have a greater weight and therefore that their influence on the pooled estimate of effect is greater and therefore any potential risk of bias will influence your estimate of effect more dramatically for quantitative analysis so yes it's important to think about and waiting no cross study by outcome thank you hmmm is there other any more questions thanks Susie there are indeed some other questions before we get to those adjacent through the chat room and evaluation form so there's a link to an evaluation form that if you could take just a couple moments to complete we'd really appreciate that we'll also be sending that link when we sends the slides but if you could take a moment to complete it we'd really appreciate your time the other questions that we had Lucy were regarding the the source of funding lisa has asked why would the first standing not be a potential source of bias and this is a very common pattern in fasting and the current guidance is that it will be picked up in other processed domains so the source of funding itself it doesn't necessarily result in a systematic error or deviation from the truth and that this will be picked up by thickness generation lack of binding or most importantly potentially selective reporting and but no itself is not and currently a factor within the risk of last tool there will be some work going on to that but I think that that's pretty hard and fast rule and it was really interesting yellow man and who who published sir Brando study this year who worked quite closely with the BMG also and took all of the risk of bias assessments from across all the copper reviews and looked after others and a huge proportion of reviews are including source of funding and another risk of bias and we qualified it within our mandate kind of put the word out there that it's not as itself an explicit source of bias or and other sort of slide but it's definitely something that you could report definitely not saying something that shouldn't be reported but it's not necessarily explicit potential for a systematic error in your estimate of effect I hope that clear then if not please let me know I'll give another explanation may be written that explanation thank you for reading that thanks very much Lucy a couple of other questions that came through there's a software named gate that is designed for assessing the risk of bias how do you feel about using that I have never had this software so you can you remove the iron what supports the software cool certainly this software is called date as in something you use to keep your horses and gate designed for our risk of bias and it's designed by professor rod Jackson rod Jackson okay I have not heard but that hasn't been done in conjunction with the the BMG and I don't believe it's inin done in conjunction with the risk of fire to developers accidentally asked Julian Jonathan the others BMG and executive members if if they've heard of it and I'll try and find out what the speakers on on this software and update everybody reserve with the blinding references spot sorry I don't know thanks very much this become missing that if you wouldn't mind sending that true to us if you could send that true to either Lucy's email or switch to the Canadian Cochrane Center we'd really appreciate that reference thank you and the last question and we have a question from the original wondering if it was a if you're wondering about grade pearl are we talking about grade Pro perhaps oh I think question wondering perhaps the software is grade I'll masonry perhaps thinking about grape rule justify justify and I believe those are all of our questions what opal Noriko dances with me in Korea assess the risk of buyers happy without grassland yeah and and many many people do that some you can just take the domains ad include them in an Excel spreadsheet for example in conjunction with your data extraction for an assessment of Isis is high low and unclear using filters and in itself quite in a quite straightforward way and also Redman is is free for everybody to use you don't have to be a Cochran author to go to the Cochran website and if you search in the search bar in on the clock and website Redman you can download downloads the web and software I'll take your studies into that and you can do your meta analysis and reduce your risk of bias figures if you want to include them in a non Cochran review and it's free friendly to you it's really good thank you very much to see I believe because that's all of our questions could I ask everybody to send through a round of applause a round of applause to thank Lucy for taking the time today to prevent this we really appreciate you being available and it would be marvelous um if you wouldn't mind responding to those questions I'd be happy to extend through any responses to anybody who joined today's webinar again we'll also be sending through a copy of the slides and we'll be sending through a link to that evaluation form again so thank you very much to everybody for participating we really appreciate you joining us and we hope you join us in a couple of weeks on December 11th for our next webinar on Prisma and the equity extension so we hope you can join us for that thank you so much for your time everybody

Leave a Reply

Your email address will not be published. Required fields are marked *