A structured 7 step process for risk assessment of a medical device clinical trial
In this essay, I discuss a systematic methodology for evaluating risk in your medical device clinical trial. This is a methodology that has proven itself in hundreds of security and privacy compliance risk assessment projects in a wide variety of healthcare, clinical and IT scenarios.
It is a given that the people charged with your clinical trial planning,regulatory affairs and operations are better at executing standard operating procedures then in performing risk analysis and thinking like attackers.
D.Lieberman
Risk assessment is a process that starts before you write the protocol, when you are writing the CRF (to determine what data to collect) and any time there are amendments to the study.
See the below graphic from the Transcelerate Web site to see why procedures do not protect your clinical trial and why SDV does not assure patient compliance to the protocol. Note that “Material risk” is any threat to the success of the study from problems with study startup to problems with poor patient compliance.
Introduction
Does counting compliance activities secure the deliverables of your clinical trial?
First define “secure”.
Security is about reducing the impact of unpredictable attacks on assets – in your case, attacks on the 2 most critical assets of your clinical trial – the data and the subjects.
Some examples of unpredictable attacks on your clinical trial:
There may be multiple sources of data errors at sites, ranging from mistakes, misunderstandings, sloppiness and all the way to incompetence.
There may data fraud – deliberate fabrication or falsification of data
There are patients that comply and patients that take their treatment randomly and in strange and wonderful ways.
There are patient reported outcomes that make sense and then there are the people who write War and Peace in the ePRO system and crash the SAS analysis program with special characters they used.
Will compliance activity check-boxing mitigate ANY of the above attacks?
No.
How to mitigate unexpected attacks on your data and patients
Once we understand that check-box compliance procedures are not a good countermeasure for threats to your study deliverables (solid scientific data, patient safety, patient compliance with the clinical protocol) what are our options for mitigation?
Consider your strengths and weaknesses.
Starting with your weaknesses, it is a given that the people charged with your clinical trial planning,regulatory affairs and operations are better at executing standard operating procedures then in performing risk analysis and thinking like attackers.
There is a fundamental divide, a metaphorical valley of death of mentality and skill sets between a regulatory-affairs and clinical operations mindset and a professional security mindset.
This essay offers a systematic approach – if you will, a common language, a language of people-centric threat modeling that helps clinical managers cross the chasm between thinking like a regulatory affairs person and thinking like an attacker who wants to destroy your study.
Start by thinking about how your study can be attacked.
Analyzing the impact of attacks on medtech studies requires hard work, hard data collection and hard analysis. It’s not a sexy, fun to use, feel-good application like Apple Music. Risk analysis may yield results that are not career enhancing, and as the threats get deeper and wider with bigger and more complex trials – so the security valley of death deepens and gets more untraversable.
There is a joke about systems programmers – they have heard that there are real users out there, actually running applications on their systems – but they know it’s only an urban legend. Like any joke, it has a grain of truth. IT and security are primarily systems and procedures-oriented instead of customer-safety oriented. Similarly – clinical regulatory affairs are primarily paper and process-oriented instead of attack-oriented.
Leave your paper and process comfort zone
If the essence of security is protecting the people who use a company’s products and services then the essence of security for a clinical trial is protecting patients and acquiring reliable data.
A structured 7-step process for risk analysis of your clinical trial
We propose a structured process for risk analysis and ongoing risk management. No previous training is required and the process can become a key part of a medtech developer’s management toolkit.
The risk analysis and management process has 7 steps as described in the below schematic (“the risk management loop”). The process uses threat modeling and quantitative risk assessment methods based on providing a financial value to assets (such as EDC systems and patient eCRF records) in order to determine value at risk and prioritize security countermeasures.
The 7 step risk process provides a systematic way to manage risk while responding to changes in regulation, business environment and clinical research feature set/functionality. Let’s start with some basic definitions:
Definitions
Vulnerability is a weakness, limitation or a defect in one or more of the system’s elements that can be exploited to disrupt the normal functionality of the system. The weakness or defect may be either in specific areas of the system, its layout, its users, operators, and/or in its policies and procedures.
Countermeasure is a technical, physical or procedural safeguard that mitigates one or more vulnerabilities.
Asset – data, systems, physical assets or intellectual property of value to the organization.
Threat – action(s) that exploit vulnerabilities in order to damage assets.
Asset value – the financial value of an asset that is destroyed of stolen. Assets may be digital (software source, physical (a server) or commercial (a corporate brand).
Damage to Asset – damage to a physical asset or damage to a digital asset in terms of breach of confidentiality, impacted system availability or broken integrity of systems and/or data. Damage is estimated in financial terms.
Threat probability is the likelihood that a threat will turn into a real attack. Threat probability can be described in terms of ARO – Annual Rate of Occurrence; i.e. how many times a year that the attack is forecasted to happen.
Threat risk is the likelihood of damage that may be caused to one or more assets by the threat.
Recommended countermeasures the possible countermeasures that reduce the threat’s risk based on the countermeasures that mitigate the threat vulnerabilities.
Actual countermeasures (aka mitigation plan) is a subset of recommended countermeasures that is assumed to be the most effective for mitigating a specific threat. Choice of specific safeguards is often a judgment call of the threat analyst.
Countermeasure cost is the financial value that is associated with the implementation of a specific countermeasure.
Countermeasure cost effectiveness is the degree of mitigation introduced by a specific countermeasure to the overall risk in the system in relation with the cost of implementing this specific countermeasure.
Attacker is a person (or group of persons) that may perform the steps of a specific threat scenario.
Attacker Types are the various classes of attackers that are differentiated according to their motivation, qualification, available attack tools and their accessibility to the attacked system’s resources.
Entry Points are points of entry made by attackers into the system, for example doors in a building or users who have a login to your EDC system.
The 7 step risk analysis loop
Risk analysis is not a one-way, one-time process you do, report and file away. Analyzing attacks and risk in your studies is an ongoing exercise always relying on quality human intel from the field – from CRCs, subjects and site monitors.
Step 1 Set scope
The threat analyst(s) will identify reasonable threat scenarios and their probability.
Read this if you are new to risk analysis
Choose one (1) question you want to answer. That’s it. Only one (1). For example – “what is the threat scenario for patients participating in the study and not passing inclusion/exclusion criteria”? After you have nailed the question, nail the threat scenario – i.e. how it can happen. After you nail the threat scenario – quantify the threat in terms of probability of occurrence and its impact and potential damage to your study.
Read this if you are a medtech developer
In a medtech study which uses wearables, connected medical devices or mobile medical device apps (or any combination thereof), having up-to-date documentation of software functionality and architecture is required in order to correctly identify vulnerabilities and threat scenarios. The following documentation is required as part of the risk analysis process:
1. Functional description of the system including relevant use cases
2. Architectural diagram of the system
3. Documentation of sub-modules
How to document the risk assessment for your medical device study
Up-to-date documentation of the study protocol and CRF is required in order to correctly identify vulnerabilities and threat scenarios. Historical records of protocol amendments is unnecessary.
The following source documentation is required as part of the risk analysis process:
Study protocol
Treatment schedule and visit flow
eCRF
CRF edit checks
These documents must be detailed enough to be used as reference for the decisions regarding the applicability of various threat scenarios to the analyzed system.
Step 2 Identify assets of your study
The correct mapping of assets (EDC database, patient safety, drug accountability data, etc), their financial value and the evaluation of financial loss to the sponsor when these assets are damaged or stolen, is one of the most critical tasks in the threat analysis process. The assets value is used as the basis for calculating threat risks and countermeasures priorities.
Asset valuation is not a one-time activity
Due to the importance of asset valuation, the asset list and corresponding values should be reviewed once a year by the controller or CEO during the course of the study.
Step 3 Identify the moving parts (components) in your study
Using a systems approach to your study, map the moving parts in your study. This will include application software components (EDC, IWRS, ePRO, centralized monitoring systems etc), people functions (study monitors, site monitors, project manager, CRCs, principal investigators).
Map the “moving part” entities to assets (for example patient records) and update the threat model with the components and functions.
Tagging different components and functions in the system help the analyst in classifying the various data and software entities and relating them to specific vulnerabilities and safeguards such as protecting PHI processed by an outsourced call center.
Step 4 Identify your study vulnerabilities
Identifying and classifying vulnerabilities requires the analyst to be intimate with the study primary and secondary endpoints, safety endpoints, protocol design, implementation and deployment details. The analyst should also be familiar with clinical operations procedures and the types of users, customers and patients that use the system or are involved with delivering services.
Step 5 Build / update the threat model
Classifying attacker types
The basic attacker types are: study user roles (site and study monitors,Pis, CRC, project managers, IT staff or cloud EDC providers) , malicious outsiders, trusted insiders and other site staff and outsourcing service providers. Additional attacker types (such as hacktivists) may be added when relevant. Different attacker types will have different motivations and different costs for mounting an attack. Attack motivation and cost are an important part in estimating threat probability since cheap attacks by highly motivated individuals are more likely than expensive attacks by attackers with little to gain.
Identifying attack entry points
The best strategy for this step is to review attacker types and document every possible way a potential attackers could access the system. The list of entry points may be refined in the course of the risk management loop.
Step 6 Build your risk mitigation plan. Calculate residual risk
Risk assessment is not over until the fat lady sings. You walk away from the risk assessment table with a much deeper understanding of what threats count and how much residual risk you have after deploying controls – technical controls, monitoring of deliverables, patient safety monitoring
This is the most important step of the risk analysis and management process. The outputs are:
A map of the relationships between threats and area tags, assets, attacker types, entry points and vulnerabilities
An evaluation of the total damage and risk parameters for each of the threats
Write mitigation plans
Calculate residual risk – i.e. how much risk exists after you implement your new controls.
Since threats are the most complex entities in the model, the process of identifying and constructing threat’s elements and parameters has a ‘decomposition’ flavor. During this step the analyst(s) will have to return to previous analysis steps in order to create missing entities, such as assets and vulnerabilities that are referenced by the threat that is constructed.
Step 7 Validate your findings
The accurate identification of countermeasures and their relations with vulnerabilities is the basis for validating the correctness of the risk mitigation plan. The best way in our experience of validating a risk analysis is to show it other people outside your office and ask them what they think.
Validation output
A list of countermeasures that mitigate vulnerabilities: The list should include the implementation cost and an indication if the countermeasure is already implemented.
A map of the relationships between countermeasures and vulnerabilities: This map shows which vulnerability is mitigated by which countermeasure(s).
A validated risk mitigation plan will include the following management level reports:
Threats ordered by risk
Threats ordered by the financial damage
Safeguards ordered by risk mitigation percentages
Safeguards ordered by their effectiveness (mitigation/implementation cost)
Asset value at risk before mitigation
Residual value at risk after the mitigation plan
Summary
We have presented a systematic 7 step process for identifying and analyzing threats to the assets of your clinical trial – whether its unpredictable user behavior or patients at risk.
Assessing the risk posture of any study will benefit from this proven systematic methodology and will help you take a paper and process-oriented study team from a place of weekly and monthly reports and activity-counting to a faster-moving, and vastly more effective place of risk understanding and mitigation.
100X faster to deviation detection in medical device studies.
