Approaches to Assay Processing Package Validation
Ellis Hughes
@ellis_hughes
Fred Hutch Cancer Research Center - SCHARP - VISC
What is Computer System Validation?
Establishing documentary evidence that your software performs a
- Procedure
- Process
- Activity
in compliance with specifications with a high degree of assurance.
Required for FDA submission
21 CFR 11
General Principles of Software Validation (FDA 2002)
Improved Quality and Safety
Results in Faster Processing
Promotes Trust
Validation
- Does the system behave as we expect
- Returns the correct value
- Throws an error when values are unexpected
Verification
- Do the outputs match between two independent approaches
Validation in Practice
Validation can be a high bar
- Labor-hours
- Tracking Progress
Tom Shadyac, et al. BRUCE ALMIGHTY. USA, 2003.
https://media1.tenor.com/images/3267a92e265348c0cc435a7989d65e23/tenor.gif
Develop and get signoff on specifications
Write Code following GPP and record function authorship
Develop Test Cases and get signoff
Write Test Code based on Test Cases
Manually evaluate and document test code results
Review documentation and get final signoffs for release
Super Mario Bros., Nintendo, 1985, NES
source: https://gfycat.com/earnestgiganticdorado
VALIDATION
+
Adam Mckay, et al. STEP BROTHERS . USA, 2008.
https://bethgittings.tumblr.com/post/34641171446
Ellis Hughes
- Statistical Programmer
- Statistical Genetics
- HIV Vaccine Research
- Community
Seattle UseR Organizer
-
Cascadia R Conf Committee
Fred Hutch and SCHARP
Blood Stem Cell Transplantation
and Immunotherapy
Cancer Risk Factors, and Causes,
Prevention and Outcomes
Vaccine Development
and Virus-Associated Cancers
Molecular Underpinnings of Cancer
Tumor-Specific Translational Research
Worldwide Impact on HIV/AIDS Vaccine Research
Package Validation
Specifications
Code
Test Cases
Test Case Code
Documentation
Package Structure
-- DESCRIPTION
-- man
-- NAMESPACE
-- R
|__functions.R
-- tests
-- vignettes
Specifications
What are the requirements of the package?
- What will the package do?
- What are the expected ouputs?
- What are the high level steps that need to be followed?
- My RStudio::Conf presentation will
- The contents will cover my teams approach to validation
- Be 15-20 minutes long
- Be entertaining
Recording specifications
Record Specifications in Rmarkdown
- 1 file per specification
- Number specifications and details
Augment Rmarkdown with Roxygen
- Specification Title
- Document who and when
#' @title Specifications For RStudio Conf 2020 Success
#' @section Last Updated By:
#' Ellis Hughes
#' @section Last Update Date:
#' 2020/01/28
+ _Specifications_
+ 1.1 Presentation must explain validation procedure.
+ 1.2 Presentation will be between 15-20 minutes long
+ 1.3 Be entertaining, sucess measures by causing at least 3 people to laugh.
+ 1.4 (Optional) Fame and Glamour and start branded accessories chain.Quickly update
- Specifications
- Ownership
Close proximity to the task
De-couple Specifications
-- vignettes
|__Validation
|__Specifications
|__specification_001.Rmd
Code
Queen - I Want To Break Free, 1984
https://imgur.com/gallery/QC6Nllw
Dont worry, I won’t be telling you how to code
Documenting code ownership
- External resource
- excel, word, smartsheet
- After the fact
- Cumbersome
#' @title Deliver Jokes
#' @description
#' Deliver jokes with punchlines. Wait 3 seconds for the punchline.
#' @param Setup Joke setup
#' @param Punchline Joke punchline
#' @examples
#' joke('To the person who stole my presentation -','I hope you do not Excel.')
#'
#' @section Last Updated By:
#' Ellis Hughes
#' @section Last Updated Date:
#' 2020/01/29
joke <- function(Setup, Punchline){
print(Setup)
Sys.sleep(3)
print(Punchline)
}
#' @title Deliver Jokes
#' @description
#' Deliver jokes with punchlines. Wait 3 seconds for the punchline.
#' @param Setup Joke setup
#' @param Punchline Joke punchline
#' @param wait how long to wait to serve the punchline
#' @examples
#' joke('To the person who stole my presentation -','I hope you do not Excel.')
#'
#' @section Last Updated By:
#' Not Ellis Hughes
#' @section Last Update Date:
#' 2020/01/30
joke <- function(Setup, Punchline, wait = 3){
print(Setup)
Sys.sleep(wait)
print(Punchline)
}-- R
|__jokes.R
-- man
|__joke.Rd
-- tests
|__testthat.R
|__testthat
|__test_joke.R
A Case for Test Cases
Draw connections between specifications and functions
Source: https://nc3t.com/wp-content/uploads/2014/10/How-to-find-innovation.jpg
Distinct from Unit Tests
A single test case can satisfy multiple specifications
Every single specification must map to at least one test case
How to Document Test Cases
- Record which Specifications are being met
- Specify the required data
- Record the steps needed without writing the code
- Detail the expectations that need to be tested and how
Documenting Test Cases
Record Test Cases in Rmarkdown
- 1 file per Set of Test Cases
- Number Test Cases
Augment Rmarkdown with Roxygen
- Test Case Title
- Record who and when
#' @title RStudio Conf 2020 Success Test Cases 001
#' @section Last Updated By:
#' Ellis Hughes
#' @section Last Update Date:
#' 2020/01/29
+ _Test Cases_
+ Setup: Create RStudio::Conf 2020 Presentation
+ T 1.1 Test that specifications 1.1, 1.2, and 1.3 are met by practicing presentation on unsuspecting co-workers
- Present to a captive audience of coworkers and later your significant other
+ T 1.1.1 Test that the presentation was informative by asking what your audience learned.
+ T 1.1.2 Time the presentation and make sure it was between 15 and 20 minutes.
+ T 1.1.3 Test that you were entertaining by counting the amount of chuckles (>3)
+ T 1.1.3.1 Alternatively, eye-rolls from your significant other (>4)Modular Test Case Documentation
- Easily shift and update
- Easily assign Test Cases
- Track updates
-- vignettes
|__Validation
|__Test_Cases
|__test_case_001.Rmd
Test Coding
Implement Test Cases
Record results
Third Party
Benefits
- Resolves interpretation errors
- documentation
- test cases
- Identify improvements
Approach
context('RStudio Conf 2020 Success')
#' @title RStudio Conf 2020 Success
#' @section Last Updated By:
#' Not Ellis Hughes
#' @section Last Update Date:
#' 2020/01/29
test_that('T1.1',{
joke_result <- joke('What do you call a fake noodle?','An Impasta')
expect_true(is_dad_joke(joke_result))
expect_true(caused_laugher(joke_result))
expect_true(embarrased_significant_other(joke_result))
})testthat
- Familar framework
- Can run and developed interactively
- Automation
Reporter Objects
- Track each test and expectation
- Reports Success
- Records Failure mode
- Easily access contents
| OK F W S | Context
| 3 | RStudio Conf 2020 Success [3.2 s]
-- Results ---------------------------------------
Duration: 3.2 s
OK: 3
Failed: 0
Warnings: 0
Skipped: 0Using a custom print function the Reporter object
Modular Test Code Documentation
- Easily update Test Code and expectations
- Track updates
-- vignettes
|__Validation
|__Test_Code
|__test_code_001.R
Final Document
What we have so far
- Documents for each Specification in markdown
- Roxygen documentation for Code
- Documents for Test Cases in markdown
- Test Case Code augmented with Roxygen documentation
Roxygen2
- Generate list of ownership of all documentation
Rmarkdown
- Record Specifications and Test Cases
- Generate Final Validation Document
Testthat
- Record results of test code
'Captain Planet and the Planeteers' Theme Song: Turner Broadcasting System
Source: https://www.syfy.com/sites/syfy/files/styles/1100xauto/public/captain-planet.gif
-- DESCRIPTION
-- man
-- NAMESPACE
-- tests
-- R
|__jokes.R
-- vignettes
|__Validation.Rmd
|__Validation.pdf
|__Validation
|__Specifications
|__specification_001.Rmd
|__Test_Cases
|__test_case_001.Rmd
|__Test_Code
|__test_code_001.R
Many thanks to all involved
Marie Vendettuoli
Anthony Williams
Jimmy Fulp
Bharathi Lakshminarayanan
Rafael Kuttner
Alicia Sato
Shannon Grant
Paul Stutzman
Kate Ostbye
Many many more people
Thank You
Presentation available at:
thebioengineer.github.io/validation
github.com/thebioengineer/validation
@ellis_hughes
Presentation brought to you by {sidescroller}
