Introduction to pilot studies: advantages and disadvantages
As Porta (2008) defined it, “a pilot study is a small-scale test of the methods and procedures to be used on a larger scale”. Pilot studies mostly occur in medicine (in particular, clinical research) and social sciences, and their major role is to collect data, as well as to prepare questions and issues related to the full-scale work. They may be especially useful in calculating the budget, determining specific project requirements, and learning about the adequate number of contributors and participant recruitment rates. (Conn et al. 2010) This article presents a basic understanding of pilot studies, their goals, as well as their ethical aspects and possible misconducts. Furthermore, we touch on the most common pilot study funding options, and debate when and how results of the work should be published.
Photo by CDC on Unsplash
Pilot studies versus feasibility studies
Large scale research projects such as clinical trials, require much time, expertise, and money. It may be challenging to find appropriate ways to properly plan them out; however, in order to optimise the use of often limited resources, such studies need to be indeed carefully designed. (Knight et al. 2019) Feasibility studies may be a part of responsible preparation, aiming to check if and/or how the planned experiment can be performed in the future, and whether the expected conclusion will be reachable (Campbell et al. 2018) A standard form of a feasibility study is a pilot project; that is, a small-scale study conducted in preparation for a larger investigation, for example, a randomised controlled trial, usually referred to as RCT (Shanyinde et al. 2011). Pilot studies should test key elements of the trial, including recruitment and retention strategies, intervention delivery, data collection methods and adherence to the study protocol. (Leon et al. 2010)
Objectives of a pilot study
The goal of a pilot study is to assess the acceptability of specific research approaches. In other words, rather than testing a hypothesis, pilot studies investigate the feasibility of a protocol and the efficacy of instruments and policies. They provide preliminary data, which helps define the research question – and, beyond that, pilot studies can test the appropriacy of the randomisation and blinding, plus the recruitment policy and the wording of the consent.Thus, such interventions allow for building knowledge on different types of studies required for the final paper, as well as selecting the most appropriate outcome measures. As mentioned above, a pilot work is also useful for estimating the time and cost of the main study.
However, it is debated whether pilot studies are suitable to serve as judgement about either safety or tolerability of a treatment.(Arain et al., 2010) It should be stressed that they also won’t necessarily provide a preliminary test of the research hypothesis. The reason for this is that, due to their smaller size, pilot studies are not suitable for statistical analyses. Hence, through a pilot work, there is no chance to fully estimate the effect sizes for power calculations of the larger scale study.
Possible outcomes of pilot studies
According to In (2017) typical interpretations of pilot studies include
- termination of the study (cannot proceed with the main study)
- can proceed with the main study after modifying the study design
- not necessary to modify the study design, however, monitoring is needed throughout the study procedures
- can proceed without modifying the study design
Ethical aspects
As usual, we also touch on the ethical aspects of our topic. It is important to bear in mind that underpowered trials do not necessarily count as pilot studies. According to Campbell et al. (2018) "Trials which use surrogate endpoints should only be described as ‘pilot’ when a definitive trial is a distinct possibility and the authors consider conditions which would indicate whether the definitive main trial was worthwhile and feasible. Simply because a trial uses a surrogate endpoint is not justification for calling it a pilot trial."
Furthermore, a fair practice will report uncertainties in all cases, and it will also avoid drawing unjustified conclusions – the results are expected to be rather descriptive. So, the key is to justify the sample size and to set the success criteria prior to the work; this even allows for research integrity; for instance, making sure participants are familiar with the nature of the project, including the possibility that a main study may not be carried out.
Funding
What kind of funding options are available for conducting pilot studies, and (why) is it worth it? Besides research grants and foundations, crowdsourcing may also be considerable; this includes voluntary participants, donated computer time, and direct transactions. Note that commercial sponsors are less common (Stachowiak, 2020), however; funding may come from relevant industrial partners, for instance from pharmaceutical manufacturers.
Publication and our practice at AKJournals
Some results of pilot studies may be generalised, and in exceptional cases, they have the potential to provide data that won't be relevant to incorporate into the final paper. (Algase, 2009) Should this be the case, publication is highly considerable. However, when, for instance, sample size (see Noordzij et al. 2010) and/or study population calculations are irrelevant, low efficiency or possibilities of future duplications limit the contribution to science,
To conclude, when carried out consciously, keeping in mind and avoiding possible misconducts, a pilot study will serve as a preliminary research. As such, it should first and foremost address methodological questions that a final study poses, from appropriateness through practical questions and data collection to concrete procedures. Whether qualitative or quantitative, the goal of pilot studies is to provide data for eliminating the previously detailed issues related to a larger work. (Algase, 2009) It is a question of responsible considerations to determine if a pilot study is proven to have the potential to be a valuable and useful research (for instance of statistical significance), that the scientific community can truly benefit from. Should the outcomes of a pilot study turn out to be exceeded once the final study is accomplished, in order to avoid drawing premature conclusions and losing the trust of investors/funders, pilot studies are better left unpublished.
Note that here at AKJournals, most titles accept high quality reports of pilot studies.
References
Algase, D. L. (2009) To publish or not? That is not the real pilot study question? Research and Theory for Nursing Practice: An International Journal. Springer. 23(2)
Arain, M., Campbell, M. J., Cooper, C. L., Lancaster, & A. G. (2010) What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology. 10(67) https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-10-67
Campbell, M. J., Lancaster, G. A., & Eldridge, S. M. (2018). A randomised controlled trial is not a pilot trial simply because it uses a surrogate endpoint. Pilot and Feasibility Studies, 4(130)
Conn, V. S., Algase, D. L., Rawl, S. M., Zerwic, J. J., & Wyman, J. F. (2010). Publishing Pilot Intervention Work. Western Journal of Nursing Research. Sage. 32(8) pp. 994–1010.
In, J. (2017) Introduction of a pilot study. Korean Journal of Anesthesiology. 70(6) pp. 601-605. https://ekja.org/upload/pdf/kjae-70-601.pdf
Knight, C. (2019) What are pilot studies and clinical trials? https://www.news-medical.net/health/What-are-Pilot-Studies-and-Clinical-Trials.aspx#1
Leon, A. C., Davis, L. L., & Kraemer, H. C. (2010). The role and interpretation of pilot studies in clinical research. Journal of Psychiatric Research. Elsevier. 45. pp. 626-629.
Noordzij, M., Tripepi, G., Dekker, F. W., Zoccali, C., Tanck, M. W., & Jager, K. J. (2010) Sample size calculations: basic principles and common pitfalls. Nephrology Dialysis Transplantation25(5) pp. 1388–1393, https://doi.org/10.1093/ndt/gfp732
Porta (2008) Dictionary of Epidemiology, 5th edition
Shanyinde, M., Pickering, M. P., & Weatherfall, M. (2011) Questions asked and answered in pilot and feasibility randomized controlled trials. BMC Medical Research Methodology. 11(117) https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-11-117
Stachowiak, J. (2020) Funding for Pilot Studies. Verywell Health. https://www.verywellhealth.com/pilot-study-research-2440747
