Invisible gorillas and detection of adverse events in medical device trials

Weekly Episode #1 - Patients and study monitors are both people.

What is easier to detect in your study – Slow-moving or fast moving deviations?

This post considers human frailty and strengths.

We recently performed a retrospective study of the efficacy of automated study monitoring in orthopedic trials. An important consideration was the ability to monitor patients who had received an implant and were on a long term follow-up program. Conceptually, monitoring small numbers of slow-moving, high-risk events is almost impossible to do manually since we miss a lot of what goes on around us, and we have no idea that we are missing so much. See the invisible gorilla experiment for an example.

One of patients in the study had received a spinal implant and was on a 6 month follow-up program dived into a pool to swim a few laps and died by drowning despite being a strong swimmer. Apparently, the pain caused by movement of the insert resulted  in loss of control and a severe adverse event. The patient had disregarded instructions regarding strenuous physical activity and the results were disastrous. 

It seems to me that better communications with the patients in the medical device study could have improved their level of awareness of safety and risk and perhaps avoided an unnecessary and tragic event.

Subjects and study monitors are both  people.

This might be a trivial observation but I am going to say it anyhow, because there are lessons to be learned by framing patients and monitors as people instead of investigation subjects and process managers. 

People are the specialists in their personal experience, the clinical operations team are the specialists in the clinical trial protocol. Let’s not forget that subjects and study monitors are both  people.

Relating to patients in a blinded study as subjects without feelings or experience is problematic. We can relate to patients in a personal way without breaking the double blinding and improve their therapeutic experience and their safety. 

We should relate to study monitors in a personal way as well, by providing them with great tools for remote monitoring and enable them to prioritize their time on important areas such as dosing violations and sites that need more training. We can use analytics of online data from the EDC, ePRO and eSource and connected medical devices in order to enhance and better utilize clinical operations teams’ expertise in process and procedure.

A ‘patient-centered’ approach to medical device clinical trials

In conditions such as Parkinsons Disease, support group meetings and online sharing are used to stay on top of medication, side effects, falls and general feeling of the patient even though the decisions on the treatment plan need to be made by an expert neurologist / principal investigator and oversight of protocol violations and adverse events is performed by the clinical operations team. There are many medical conditions where patients can benefit by taking a more involved role in the study. One common example is carpal tunnel syndrome. 

According to the findings of an August 3rd, 2011 issue of the Journal of Bone and Joint Surgery (JBJS), patients receiving treatment for carpal tunnel syndrome (CTS) prefer to play a more collaborative role when it comes to making decisions about their medical or surgical care. 

Treatment of carpal-tunnel syndrome which is very common and also extremely dependent upon patient behavior and compliance is a great example of the effectiveness of “shared decision-making, or collaborative, model” in medicine, in which the physician and patient make the decision together and exchange medical and other information related to the patient’s health.

As the article in JBJS concludes:

“This study shows the majority of patients wanted to share decision-making with their physicians, and patients should feel comfortable asking questions and expressing their preferences regarding care. Patient-centered care emphasizes the incorporation of individual styles of decision making to provide a more patient-centered consultation,” Dr. Gong added. 

In a ‘patient-centered’ approach to medical device clinical trials, patients’ cultural traditions, personal preferences and values, family situations, social circumstances and lifestyles are considered in the decision-making process.

Automated patient compliance monitoring with tools such as are a great way to create a feedback loop of medical device clinical data collection,  risk signatures improvement, detection of critical signals and communications of information to patients. Conversely, automated real-time patient compliance monitoring is a a great way of enhancing clinical operations team expertise.

Patients and study monitors are both people. 

Millennials are the future of clinical trial data management

esource tp get smart to market

Millennials, born between 1980 and 2000 and the first native generation of the digital age, are the quickly approaching additions to the modern workforce. Regardless of whether private or public sector Millennials are soon to become the bulk of the global workforce.

At present, Millennials represent 34% of the current US workforce (up 9% from 25% in 2015), and by 2020 50% of workers will be of the Millennial generation. As the demographics of present job seekers continues to shift, companies need to adjust their culture, facilities and technology to cater to the new generation.

Regarding the clinical trial industry, Millennials are not only the next generation of data managers and monitors, but will soon make up the bulk of the study subjects as well.

Choosing the right tool and UX for millennial subjects becomes acute considering usability factors and patient compliance issues for people under 30.


A powerful alternative to checklists for assuring patient compliance

medical device clinical trials

Danny Lieberman, founder and CEO of the leading cloud provider of
clinical compliance as a service, talks about breaking out of a patient compliance
checklist mentality by starting with one question.

The 3 pillars of GCP (good clinical practice)

1. Patient safety
2. Protocol compliance
3. Data quality

(We note that setting the focus on the primary clinical and safety end-points results in formulation of GCP as an exercise in optimizing patient compliance to the protocol.)

With the understanding that clinical trial site monitors commonly use checklists for their site visits our first question is to challenge the utility of checklists:

To what extent do fixed checklists enable the study monitor and sponsor
to assess the impact that study deviations have on protocol compliance?

Take for example the activity of monitoring IC (informed consent); a best practice informed
consent monitoring checklist looka like this:

Informed consent monitoring checklist

1. Was the consent form used, and translated versions, approved by the IRB?
2. Was the ICF the most current and approved version?
3. If the consent is available in more than one language, was the participant given a chance to choose the language he/she prefers?
4. Did the participant receive full explanation of the contents of the ICF?
5. Did the participant have ample time to ask any questions and were they addressed adequately? Was the ICF signed before any study procedures? (N/A if the trial has received an exemption from IRB to consent after some study procedures)
6. If the subject is unable to read, was an independent witness present throughout the consent process?
7. Was the participant coerced?
8. Did the participant apparently understand the contents of the ICF?
9. Was IC form signed appropriately?
10. Was the environment suitable for the IC process?
(Courtesy of Global Health Trials)

You can check-off 11 items on the list but there is only 1 question that matters:

“Are there patients participating now in the study who did not sign the ICF (informed consent form)”

Why does the version or the environment matter if the patient is enrolled without informed consent? How does this checklist evaluate the impact of deviations? Does the checklist provide any quantitative measures of patient compliance?

After you ask that 1 question – (Are are any patients enrolled who did not sign ICF?) you can go on to quantify the impact (by asking how many patients are enrolled in the study without signed ICF) and then proceed to provide corrective and preventive actions.

In this article we suggest considering an alternative approach based on generating and analyzing multiple threat scenarios for the clinical study being monitored.

Since clinical trial data is highly-dimensional (typically 500-1000+ dimensions) we may reap significant benefits from this approach since with so many dimensions there tend to be many unconnected and undiscovered stovepipes of compliance and data governance.

Multiple threat scenarios enable auditors and study monitors to side-step large scale self-assessment checklists and problematic integration of data across stovepipes (large drug studies and large CROS like Quintiles, PPD and ICON typically use multiple systems from multiple vendors creating multiple unconnected stovepipes of data – one of the key reasons it takes 5-7 weeks to respond to a deviation) and focus on key assets, attacks and common vulnerabilities in key operational processes of the clinical trial like informed consent, eligibility criteria and treatment compliance (whether treatment is self-administered by the patient or administered by medical staff in a hospital).

In our experience, the sponsor is primarily interested in how cheaply the audit can be done and how much time and money they can save further down the road. For the  business unit developing the medical device or drug, using a technique of multiple threat analysis will help show the best and most cost-effective way to progress from audit to patient compliance.

Do you base your regulatory affairs policy on Google?

You can do some homework online and then hire a clinical regulatory and compliance consultant who will walk you through the various GCP requirements and help you implement as many items as possible. This seems like a reasonable approach, but the more controls you implement, the more money you spend and moreover, you do not necessarily know if your risk posture has improved since you have not examined your value at risk – i.e how much money it will cost you for rework if more patients have to be enrolled due to non-adherence to protocol.  Recall that patient protocol compliance is central to the success of your clinical trial and the defense of your claims with the FDA should rely on your experimental design, data and risk-analysis and not on the percentage SDV (source document verification) that study monitors performed.

Top-down risk-analysis

Taking a page out of the privacy and security playbook, we want to do a top-down risk analysis, and then continue with risk management and periodic protocol compliance activity review during the course of the clinical trial.

The best way to do that top down risk analysis is to build probable threat scenarios – considering what could go wrong – sites doing shoddy data entry or a hacker sniffing the hospital wired LAN for PHI and destroying the integrity of your randomized controlled trial.

Threat scenarios as an alternative to compliance check lists

When we perform a software security assessment of a medical device or healthcare system, we think in terms of “threat scenarios” or “attack scenarios”, and the result of that thinking manifests itself in planning, penetration testing, security countermeasures, and follow-up for compliance. The threat scenarios are not “one size fits all”.

The threat scenarios for clinical trials for AIDS diagnostics using medical devices that automatically scan and analyze blood samples, or an Army hospital using a networked brain scanning device to diagnose soldiers with head injuries, or an implanted cardiac device with mobile connectivity or immunotherapy treatment for cancer are all totally different.

We evaluate the medical device / investigational product from an attacker point of view, then from the management team point of view, and then recommend specific cost-effective, security countermeasures to mitigate the damage from the most likely attacks.

In our experience, building a risk control portfolio based on attack scenarios has 3 clear benefits;

1. A robust, cost-effective monitoring portfolio based on attack analysis results in robust compliance over time since you now have a formal methodology for evaluating new emerging issues such as mobile devices or changes to regulation.
2. Executives related well to the concepts of threat modeling / attack analysis. Competing, understanding the value of their assets, taking risks and protecting themselves from attackers is really, at the end of the day why executives get the big bucks.
3. Threat scenarios are a common language between IT, clinical operations teams and the business area managers. This last benefit is extremely important in your organization, since business delegates compliance to regulatory affairs and regulatory affairs delegates assessment to the site monitor teams and there is clearly a disconnect by the time you go from a business manager to a CRA.

As I wrote in a previous essay “The valley of death between IT and security“, there is a fundamental disconnect between IT operations (built on maintaining predictable business processes) and security operations (built on mitigating vulnerabilities).

The disconnect between sponsor business management and site monitors.

Business executives delegate clinical operations to VP Clinical who delegates to CROs who delegate compliance to sites on the tacit assumption that each are the experts in their own particular domain. This is a necessary but not sufficient condition.

In the current environment of rapidly evolving types of attacks (hacktivisim, nation-state attacks, credit card attacks mounted by organized crime, script kiddies, competitors and malicious insiders and more…), it is essential that business managers, sites and regulatory affairs professionals, communicate effectively regarding the types of attacks that their organization may face and what is the potential business impact on the clinical trial.

If you have any doubt about the importance of sponsors sharing data with sites, consider that leading up to 9/11, the CIA had intelligence on Al Qaeda terrorists and the FBI investigated people taking flying lessons, but no one asked the question why Arabs were learning to fly planes but not land them.

With fundamental disconnects between 3 key stakeholders of clinical data (sites, monitors and sponsors), it is no wonder that organizations are having difficult assessing GCP compliance in a timely fashion –

Sponsors, monitors and sites (and increasingly patients) need a common language to execute their mission, and I submit that building risk control portfolio de your clinical trial around most likely threat scenarios from an attacker perspective is the best way to cross that valley of death.

There seems to be a tacit assumption with pharma and medtech executives that regulatory compliance is already a common language of compliance for a clinical trial, but as we demonstrated at the beginning of this article, compliance checklists like ICF monitoring etc, are a dangerous replacement for not thinking through the most likely threats to your clinical trials.

Let me illustrate why compliance checklists are not the common language we need by taking an example from another compliance area – credit cards.

PCI DSS 2.0 has an obsessive preoccupation with anti-virus. It does not matter if you have a 16 quad-core Linux database server that is not attached the Internet with no removable device nor Windows connectivity.

PCI DSS 2.0 wants you to install an anti-virus and open the server up to the Internet for the daily anti-virus signature updates. This is an example of a compliance control policy that is not rooted in a probable threat scenario that creates additional vulnerabilities for the business.

Consider some deeper ramifications of check-list-based compliance to the protocol.

When a QSA or HIPAA auditor records an encounter with a customer, he records the planning, penetration testing, controls, and follow-up, not under a threat scenario, but under a control item (like access control). The next auditor that reviews the compliance posture of the business needs to read about the planning, testing, controls, and follow-up and then reverse-engineer the process to arrive at which threats are exploiting which vulnerabilities.

In the cyber security space, actors such as government agencies (DHS for example) and security researchers go through the same process. They all have their own methods of churning through the planning, test results, controls, and follow-up, to reverse-engineer the data in order to arrive at which threats are exploiting which vulnerabilities.

This ongoing process of “reverse-engineering” is the root cause for a series of additional problems:

1. Lack of overview of the the threats and vulnerabilities to clinical trials that really count.
2. No sufficient connection to best practice controls, no indication on which controls to follow or which have been followed.
3. No connection between controls and protocol deviation events, except circumstantial.
4. No ability to detect and warn for negative interactions between controls (for example – edit checks that generate large number of queries on every field, hobbling the ability of the sites to collect data in a timely manner).
5. No archiving or demoting of less important and solved threat scenarios (since the checklists are control based).
6. Lack of overview of compliance status of a particular site, only a series of historical observations disclosed or not disclosed. (Is Bank of America getting better at data security or worse? Is the Department of Clinical Neuropathology at King’s College Hospital getting better at GCP compliance or worse?)

7. An excess of paper documents that cannot possibly be read by the regulatory and clinical affairs manager at every encounter.

8. Regulatory and data borders are hard to define since the border definitions are networks, systems and applications not

Beyond checklists – using value at risk to assess impact of patient compliance violations

Checklists are good for ensuring a repeatable process but threats to your study are rooted in unforeseen events like patients without informed consent. Your threat scenarios should consider your study assets (your data, systems, management attention, reputation) values, vulnerabilities, threats and effective security countermeasures.

Threat analysis as a methodology for monitoring your clinical trial does not count activities like site visits and SDV. It is a systematic way to help you consider the fastest and most cost-effective way to reduce your risks of protocol non-compliance, safety and data quality.

10 ways to reduce clinical trial risk and they are all free

Are the lights on but no one home in your medical device clinical trial?

Collecting low-quality data means that your trial is likely to fail. You will not be able to prove or disprove the scientific hypothesis of your medical device clinical trial. You will have wasted your time.

You cannot outsource quality, you have to build it into the trial design


The 2 most common mistakes in Clinical Research Data Management

medical device clinical trials

Another survey piece that David wrote about common mistakes in clinical data management and some basic controls to stay from the common issues.

Use automated monitoring to empower people

We are all human. As much as we would like to rely on technology to automate every sector of clinical trial research, we still need the human component to study, assess data, store it and make decisions based on captured research data. We cannot remove human study monitors and data managers entirely out of the equation.


The best alternative to paper in medical device clinical trials

There is an urban legend that paper is cheaper than EDC

$1000/subject for paper-based data management (the going rate in Israel)  is a lucrative business for small CROs, independent data managers and biostatisticians, but $1000/subject is not the same as “total cost of ownership” or TCO.

The TCO of doing a clinical trial for an innovative medical device vendor will include the time spent by the scientific staff  preparing and reducing data, additional time by the data manager to clean the data and a large intangible cost of the delay to receive management reports of patient compliance, typically 2-3 months, if you are running a multi-site study with paper.

But beyond TCO, the most significant factor in a medical device vendor decision process is how fast can  you get actionable intelligence on your patients and sites and CRO?


Dates: the silent death in medical device clinical trials

Bad Dates: Assessing and assuring high quality dates in clinical trials


Clinical trials are based on collections of time-based clinical data. If the dates and time-stamps in the data set are low quality, everything else will be low quality: measurement of study progress, enforcement of visit protocols and study schedules, measurement of site progress and any clinical parameter that is a function of time, such as cumulative dosing, pregnancy and hundreds of other time-based use cases.

Jenya talks about bad dates and how really bad quality dates that can spell disaster for your clinical trial – and suggests what do to about it.


An attack modeling approach to medical device clinical trials

an attack modeling approach to clinical trial risk analysis
What does taking off your shoes and belt in the airport have in common with risk assessment in clinical trials?

Today we talk about the drawbacks of traditional risk assessment and propose an alternative approach to clinical trial risk assessment that is based on data and considering plausible attacks on your trial as opposed to fixed protocols and human monitoring processes.


5 ways to reduce costs and accelerate medical device clinical trials

Patient compliance automation

Time is money

The economic model of a multi-center clinical trial is based on a commercial biotech/biomed/pharma company sponsoring and funding clinical research in order to prove efficacy and or safety of a new drug or medical device.

In fact, although the sponsor has the most at stake by far, in clinical trials, sponsors and sites both have a clear economic interest in productivity.

Looking at money – as a rough yardstick of your clinical trial budget – 1/3 will go to the investigative sites, 1/3 to the patients and 1/3 for clinical monitoring.

1/3 for clinical trial monitoring is a lot of money.  How do you optimize that and make the investment cost-effective and time-effective while ensuring a high level of patient compliance to the study protocol?

5 critical factors in choosing software for clinical data management

Electronic data capture (EDC) systems are on track to bloom into the future norm for clinical trial data management. To put it simply, the past process of using a paper case report form (CRF) and then uploading it into an online database is needlessly inefficient, being that cloud-based services, such as EDC, are becoming the standard for data collection and storage. Data collected in a study can now be directly electronically entered into a desktop, laptop, tablet, even a smartphone.

However, as a newer technology, it is still going to take some time until project managers and clinicians will become familiar with the proper implementation of an EDC system, and time for software developers to streamline their software for rapid onboarding during its implementation.

Perhaps the project manager has failed to address required operational changes when switching to EDC from a paper-based data collection system. Or, maybe they simply choose an improper EDC system for the study they are planning to conduct. Whatever the reason, choosing the appropriate EDC for your clinical trial studies is necessary in order for this technology to be as effective as it was designed to be.

That being said, the EDC industry is gaining traction in the clinical trial market, not because it is “green” technology (bye-bye, paper!), but because when used in proper practice, it boosts a study’s efficiency (cloud-based data can be collected, monitored, and edited anytime/anywhere) and significantly cuts study costs, among other benefits.

Schematic of the Graeco Latin Square Design(source:

In the above study, the use of EDC is a smarter way to capture data, and results in fewer mistakes than paper-based methods. If you are still on the fence of whether or not to incorporate EDC into your clinical trials, take into account the basic benefits of EDC:

– Cost savings through enhanced efficiency

– Improved data accuracy & organization

– Decrease in compliance errors

– Greater data security

– Availability of remote study data access

– Outsourcing potential to further reduce costs

If you are not intimately familiar with EDC, or are looking for a new EDC system vendor, let us go over some of the considerations you should be making when you are hunting down the ideal EDC system to use in your clinical trial(s) data capture and management. There are literally several dozen EDC vendors as of today, so these tips will help you identify which vendor will suit your study.

1. Necessity

Before shopping for a vendor, first identify the specific factors and parameters your clinical trial is going to be researching and measuring. Some studies are short-term, and may only be measuring, for example, a drug’s impact on a participant’s weight. Other trials may be much more intensive, studying hundreds of patients and dozens of variables and parameters.

When looking for your ideal EDC system, account for necessity. There is no need to waste study funds on a well-known, but costly and complex EDC system if your next clinical trial does not warrant such a product.

2. Ease of use

This may seem like a no-brainer, but in line with necessity, EDC software comes in all shapes and sizes of various degrees of complexity, so to suit every imaginable clinical trial, EDC systems can be rather complicated. However, despite complexity, an intuitive user interface will lessen the time spent on on-boarding your team to learn how to effectively navigate the EDC.

It is essential that your EDC system be simple to understand for every person authorized to use it, including monitors, project managers, data managers, even patients if your study involves patient submitted data (another clinical trial component facilitated by using EDC).

When vetting vendors, do not simply rely on reviews and online testimonials. Ask the vendor for a trial run of their EDC offerings, and make certain that it not only suits your needs, but is also easily navigable.

3. FDA compliance

This only applies to clinical trials that gather and deliver data to the FDA, but since many do, it needs to be covered. If your company fits into this category, it is imperative that the EDC you are considering meets the FDA’s 21 CFR Part 11 regulations.

Do not simply take the vendor’s word for it, but research whether it in fact meets current standards, that their software is not outdated, and that there are tools in the software to rapidly make adjustments to any future changes in compliance standards. In the case of compliance, it is the vendor’s responsibility to ensure that the software meets 21 CFR, but it is the sponsor’s responsibility to ask in the first place.

4. Timeline for setup and implementation

Remember how EDC cuts study costs and enhances efficiency? That is only true if it does not take a lifetime for your system to get up and running. After you have addressed the preceding considerations, be direct with your vendor. Tell them how much time you have planned for to implement an EDC system for your upcoming clinical trial, and ask whether there system will be able to accommodate your timeline.

5. User training resources

One consideration that is all too easy to forget, is to inquire into the training and help desk provisions the EDC vendor has. Looking into these issues:

  • Do they have an EDC training program beyond a user manual for their hardware/software?
  • Is the training free, or does it require a fee?
  • How quickly is the sponsor, in fact, every user, able to get in touch with the help desk?

No matter how thorough the software on-boarding is, adjustments will eventually need to be made to the software, and mistakes do happen. Your study should have personnel appointed to be an expert in the software design, and can save you time by solving problems in-house rather than waiting to hear back from the vendor themselves. Yet, in some cases, they will not have the knowledge of how to execute a certain modification, or may have forgotten how to perform an uncommon adjustment, so they should also be able to swiftly contact the vendor for expert advice.

Navigating the EDC maze

The cloud  EDC space is flooded with vendors, some very simple, some very capable and some very complex. It can be confusing if you are new to EDC how to choose the appropriate software for your particular clinical trial. By using the aforementioned considerations, you will quickly find a suitable software vendor.

Also, make sure that you do not rush. Allocate time to follow each step thoroughly, and to completion before moving forward. The time you invest prior to implementation will save you time on your study.