Forum:Fda Sends A Warning Letter To 23Andme - Personal Genomics Service Marketing To Be Discontinued
7
21
Entering edit mode
11.1 years ago

There has been an interesting development in the world of personal genomics. The FDA sent what may be interpreted a cease and desists order:

Therefore, 23andMe must immediately discontinue marketing the Personal Genome Service (PGS) until such time as it receives FDA marketing authorization for the device.

Read more here: http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2013/ucm376296.htm

This seems to put bioinformaticians at the forefront of deciding what constitutes credible evidence that the PGS does indeed work as advertised.

personal-genomics 23andMe • 17k views
ADD COMMENT
2
Entering edit mode

It'll be interesting to see if 23andMe posts anything in reply to this. From the one side of the story that we do get, it sounds like they haven't exactly done their due-diligence.

ADD REPLY
0
Entering edit mode

well, they may just refer back to all of us ... for what is worth we are part of the community that states that this works, they just use it in the context of marketing and reaching out to a larger public beyond scientists.

It makes you wonder will there (should there) ever be an FDA request to retract a scientific paper?

ADD REPLY
6
Entering edit mode

I don't think the question is whether it works, but whether it's responsible to be giving out all this sensitive genetic data to scientifically-illiterate people without advice from a clinician.

ADD REPLY
7
Entering edit mode

Clinicians and genetic counselors are certainly an important part of the equation, but I'm leery of any regulation that says I can't have access to the information inside of my own cells.

ADD REPLY
5
Entering edit mode

I guess I'm just echoing Emily's response, but I think there's a big difference between researcher-level and direct-to-consumer level confidence. For many things, there are just too many unknowns and consumers are just too prone to misunderstanding the results. One difference to what Emily wrote is that I would want consultation with a geneticist rather than a clinician. The average clinician simply doesn't know enough to provide a patient with insight (that's not a slam against clinicians, it's not in their training).

ADD REPLY
2
Entering edit mode

I would probably say a clinical geneticist. I want them to know their stuff, but I also want them to have the background in dealing with patients and explaining the clinical options that may be associated with any genetic discovery. The same kind of people who you'd speak to if you had a BRCA2 test.

ADD REPLY
0
Entering edit mode

Guess my thoughts are the same then!

ADD REPLY
1
Entering edit mode

It is not up to the government to regulate stupidity. If a consumer can't understand what is the product he is buying, he shouldn't buy it. Let's face it, it's not like we will start seeing mass suicides...

ADD REPLY
1
Entering edit mode

Would you say the same about any medication or medical device? Following this logic you are indeed opting for closing FDA or similar agencies in other countries, but remember this could be last saving grace for many users. Good luck, should you ever need a defibrillator which is unapproved... Imo, governments do not need to regulate stupidity, they need to prevent charlatanry and fraud in the most dire circumstances. The consumer is often the weaker part in the relation to the medical industry, therefore the requirement for evidence in favor of any new medical appliance is totally justified imho.

ADD REPLY
1
Entering edit mode

You should not say what I am opting for (i.e. putting words into my mouth), as I may have a different vision for the role of FDA, which does not include it being closed.

FDA's job is to make sure that medication, medical device or otherwise health related products are correctly described and advertised by their manufacturers. It is their job to make sure the warnings on the label are real, and not downplayed. To sum it up, it is the FDA's job to make sure, that a potential user has all the necessary information he needs to make a call whether he wants this medication or medical device or whatever.

Now where we draw the line on how much information we are giving to users, and how much do we dumb it down so the average citizen understands it, that is also FDA's job. However, I do not believe it is their job to close companies down and restrict access to services, devices or medication. I am not talking about illegal stuff like drugs, illegality of drugs is another department.

It is NOT the FDA's job to enforce smarts and education on people. If someone is about to die in 1 week and wants to take a drug that could kill him today, but might save his life with a probability of p=0.01, than it's his call. Maybe kids will start paying more attention in class if they are convinced they will actually need it later on.

ADD REPLY
1
Entering edit mode

FDA's job is to make sure that medication, medical device or otherwise health related products are correctly described and advertised by their manufacturers

Well from my interpretation of the warning letter, that's exactly what they are requesting.

ADD REPLY
1
Entering edit mode

I am sorry but seemingly you haven't even read the warning letter correctly. It is not about closing down, but all about marketing and advertising in my understanding.

ADD REPLY
0
Entering edit mode

They are asking them to pull the product off the shelf because it's suggested potential by "23andMe" falls under a category that needs pre-approval by the FDA. The FDA should not ban the product, but rather give a valid description of their product that would not require FDA approval.

ADD REPLY
0
Entering edit mode

Giving people "all the necessary information" is useless if it's not in a form they can comprehend, which is where the "enforce smarts on people" part comes in. Getting the marketing guys to work on the sales-pitch and then burying the important caveats in statistics-jargon (yet providing it!) is usually a terrible idea.

ADD REPLY
1
Entering edit mode

The worst part is journalists' take home messages will be something like this:
"The problem? 23andMe's OTC kits may be just as likely to report false positives or false negatives, according to the FDA."
http://www.washingtonpost.com/blogs/the-switch/wp/2013/11/25/fda-this-dna-testing-company-could-kill-you/

I really don't see anything in the FDA's letter to justify that quote. FDA is saying, I believe correctly, they haven't gotten any evidence to the contrary. I guess you could spin that in this way, but still sounds like FUD to me.

And the article title could go in the books as the definition of FUD:
"FDA: This DNA testing company could kill you"

ADD REPLY
0
Entering edit mode

that title is definitely among lowest form of sensationalism

ADD REPLY
0
Entering edit mode

I appreciate it!

ADD REPLY
0
Entering edit mode

Now that 23andme is back in action anyone who is interested in using their raw data for more information can try out this tool: https://www.xcode.life/product/ftdna-ancestrydna-23andme-raw-data-interpretation-analysis-tools

ADD REPLY
8
Entering edit mode
11.1 years ago

My deadly disease was just a bug

Yes, I really had two mutations. But they weren't on the same gene, but on two different genes. By rare chance, both of these mutations are statistically linked to LGMD, but to two different versions of LGMD. So I didn't have a homozygous mutation, but two unrelated heterozygous ones. The web programmer at 23andme had added those two mutations together into one homozygous one in their code. And so the algorithm switched to red alert.

ADD COMMENT
2
Entering edit mode

The fact that it turned out to just be a bug is one thing, it's pretty bad but they fixed it, but even if it had turned out to be true, it's still horrific. Imagine being sent an email saying, "Hey, you're going to get progressively disabled then die young," and no further information about what the condition is and how it's going to affect you, no kind sympathetic face offering you tissues and advice and options. This guy did his research and turned out to be fine, and that's great, but how many people fall into deep depression on getting this news? How many people kill themselves? You can't give out this kind of potentially devastating life-altering news in an email.

ADD REPLY
1
Entering edit mode

For some people, I agree this may not be the best way to communicate results. This is probably why 23andMe adds an additional step for viewing results like this that are not present for non-medical and non-predictive results. If you aren't prepared to view results on-line, then I would probably recommend either not getting a 23andMe profile, not viewing that portion of the results, and/or contacting a genetic counselor to review the results with you. For example, my 23andMe report indicates that I am a carrier for cystic fibrosis and they provide a link on how to talk to a genetic counselor on that page:

https://www.23andme.com/you/genetic_counseling/

That said, I think the concern overall is an over-reaction for the following reasons:

1) There have been several publications showing that most people have no problem responding to DTC genetic testing. I can't list all the publications off the top of my head, but here is a summary of one such article:

http://www.nature.com/news/2011/110112/full/news.2011.12.html

2) In general, the accuracy for the DNA sequencing portion of the tests (currently via an Illumina SNP array) is pretty good. For example, the FDA has recently approved Illumina sequencing for clinical application. I'm also pretty sure 23andMe has checked the accuracy of the array by comparing normal 23andMe clients to the results from people who participated in the exome sequencing pilot. That said, there is a difference between the interpretation for the carrier status results (which is relatively straightforward) and all of the other results, and my understanding is that failure to communicate these results to the FDA is one of the legitimate complaints from the FDA letter.

3) I think people need to be careful and critical in all cases. For example, let's say I had a wife who was also a cystic fibrosis carrier (identified via 23andMe) and we were thinking about kids. The first thing I would do is verify the result. For example, I could order a Counsyl test from a doctor (which might be a good alternative for some people instead of 23andMe, although I think you might still be viewing your results on-line) to verify that we were in fact both carriers. I wouldn't immediately run to an in vitro fertilization clinic. Plus, medical professionals can make wrong calls too. This has certainly happened to me, which at least one time delayed hospitalization for a very serious infection. This is not an attack on the medical establishment: there is a reason I went to see the doctors in the first place. However, I think the actions by this this individual were spot on, regardless of whether something is FDA-approved and regardless of whether a result comes from a person or a computer: if something doesn't sound right, you should look into a second opinion, independent research, etc.

ADD REPLY
4
Entering edit mode

I don't think the accuracy of the genotypes is in question. It's the accuracy of the medical analysis, or its utility.

The public has been told about how the genome was going to magically unlock the secrets of human biology for a decade. I do not know that most of them realize the gigantic knowledge gap between knowing one's DNA sequence, and drawing sound medical conclusions from it, and I don't think it's exactly in 23andMe's interest to enlighten anyone on that point. I know it, and you know it, and sure, the user base of 23andMe is a little better educated than the public at large, but not that much more.

ADD REPLY
0
Entering edit mode

However, you should make sure to read the entire article:

My inherited condition page now says: “Has multiple mutations linked to limb-girdle muscle dystrophy, but they are in different genes. A person with such mutations typically does not have the condition, but can pass the mutations to offspring. May have other mutations linked to limb-girdle muscular dystrophy (not reported here).”

Once the bug was reported, it was fixed and the report was updated

ADD REPLY
3
Entering edit mode

I did read the article. The issue isn't the bug. The issue is that 23andme is offering a product while making claims about how customers can use results for improved medical care. Here, web developers at 23andme turned out to be the ones writing the scripts that programmatically offered false ( * ) clinical information - not medical professionals. Web developers are not medical professionals and should not be tasked with offering medical services. ( * - In this case, devastating, potentially life-changing and false clinical information.)

ADD REPLY
0
Entering edit mode

I agree that communication with customers is important and some customers may not have a good sense of what it is like to be part of a research project. I could totally believe this is something 23andMe needs to work on.

However, I think the best solution is not to get rid of 23andMe, but rather help improve communication regarding the confidence of results. For example, I wrote a blog post about one possible solution after a previous FDA warning was issued:

http://cdwscience.blogspot.com/2010/08/benefits-to-3-tier-system-for-dtc.html

Also, I don't think this is a typical result. For example, here is a link to my results as well an article from Lifehacker (from someone with much less experience with genomics research). I'm sure I've seen more, but these are what I could think of off the top of my head.

http://cdwscience.blogspot.com/2011/02/thoughts-on-my-23andme-results.html

http://lifehacker.com/5802559/how-to-decode-your-dna-with-personal-genomics-service-23andme

Plus, I think an unfortunate reality is that this sort of thing will happen from time to time. I think pretty much all diagnostics will suffer from some degree of false positives, false negatives, and/or human error. I know I constantly have to update the bioinformatic programs that I design (for what I would call "research grade" analysis) - especially when hunting down bugs that are only apparent when analyzing a small number of data sets.

As a different type of example, I found that there will probably need to be future modifications to the collection protocol and/or analysis pipeline from the American Gut project (when I compared by "official" report to analysis of my raw data):

http://cdwscience.blogspot.com/2013/11/my-american-gut-individual-report.html

ADD REPLY
0
Entering edit mode

Maybe I am mistaken, but isn't "Yes, we definitely need to work on our communication with the customers." a standard phrase of excuse for everything?

ADD REPLY
0
Entering edit mode

Please. I have it on good authority that no "web developers" were involved in this unfortunate, yet quickly resolved, issue. Everything that you (used to) see in the 23andMe Health section is data driven, and the said data is compiled and QC'ed by people who really know what they are doing. I would trust those people more than an average medical professional, for sure.

ADD REPLY
5
Entering edit mode
11.0 years ago
amcrisan ▴ 370

Regulators and insurers have already told us what we need for provide for PGS devices, but most people are not aware of it. What is necessary are at least these two things:

  1. Analytical Validity
  2. Clinical Validity

23andMe was dinged for not having either of these two pieces of evidence. Analytical validity evaluates the precision and accuracy of your assay AND your informatics tools. Clinical validity is showing that the marker actually distinguishes, says, BRCA positive patients from negative.

I believe every product on the market should have verified analytical validity that goes BEYOND what CLIA expects. Clinical validity for something like PGS is harder to do. Right now regulators are depending upon evidence drawn from drug development and expecting to see the same kind of thing for PGS devices. But this isn't realistic as PGS offers different kind of, clinically relevant, information. But it's not up to bioinformaticians to decide upon clinical validity either, in fact we need to co-operate with statisticians to propose reasonable studies and level of evidence of these new types of devices - this is starting, but not a big movement.

If you want to know what FDA acceptable evidence is, look at OncotypeDx Breast articles, they have all this levels of evidence the FDA expects, and also check out these PLoS collections: http://currents.plos.org/genomictests/. The scope of these examples still isn't the same as a PGS like what 23andMe offers, but again, this is what the FDA understands.

ADD COMMENT
1
Entering edit mode

This is a spot-on comment, and captures the heart of the matter. I think many of us with a foot in the clinical side of the world regard the FDA's actions on this as "of course" and aren't surprised at all. The rules are pretty clear when it comes to putting a genomic testing product out there for clinical use.

ADD REPLY
0
Entering edit mode

It's true. Even though the rules need to be a different for PGS compared to drug development, I get the impression that the FDA (with all it's huge girth) as well as other regulators are trying, hard, to develop better standards for PGS. They want to see these things succeed, because they are perfectly aware of the demand. It's difficult because the translational pipeline for PGS is sooo multi-disciplinary and so few institutions and researchers have access to these pipeline to get a meaningful dialogue going on what needs to change.

ADD REPLY
0
Entering edit mode

I agree. It might also be worth mentioning that there are ways to further limit false negatives and false positives (from a sequencing standpoint, which I think is the most important thing for well established markers like BRCA):

1) False Positives: If you see a result that would influence a medical decision, seek validation for that finding. Many customers (including myself) probably won't even need this step. I kind of hope that generic DTC Sanger sequencing companies pop up to assist with this (to decrease validation costs), but I don't know how likely this really is

2) False Positives and False Negatives: Get sequencing results for multiple family members (which I would argue makes things more interesting anyways). Random errors are unlikely to appear in multiple family members. Although the mutation could be novel, a negative result in both your mother and father is probably a strong indication that your positive result is not true (and vice versa).

3) False Negatives: Weaker than suggestions #1 or #2, but it is important to keep traditional risk factors in mind. If the genetic test provides a weaker risk than you expected, maybe you should double-check the result. For example, if you have other family members with BRCA mutations (or at least breast and/or ovarian cancer), it might be worth validating a negative result.

ADD REPLY
1
Entering edit mode

I would tend to agree with this, but needing to run multiple genomic tests to validate a finding, even an important one, has serious limitations that could damage the reputation of PGS.

For example, physicians use Onctotype Dx breast cancer test to determine whether or not a patient should have chemotherapy with their tamoxifen treatment. They use ONE genomics test to make the evaluation. The evaluation is made in concert with other factors like the patient's health status, longevity, and so on, but they don't order multiple genomic tests. In fact multiple genomics tests don't exist. If there were multiple genomic tests and they give contradictory results how does a physician evaluate this? Physicians have limited time and need to know how to make their decision with an accurate one-stop-shop; this is very valuable and what the medical community is looking for.

If 23andMe could prove it is such a one-stop-shop (which it can't) it's an immensely value device. But if I flip a coin, tell you that you have cancer, which causes you to follow-up on that to validate it, my question is : what is the point of the genomic test if you can place limited confidence on the accuracy?

ADD REPLY
0
Entering edit mode

If you want to try and look at your genome sequence, I think there will always be some errors. However, the error rate should be relatively small (with the SNP array, the error rate should be <0.01%).

If given a choice between looking at a genome sequence with some errors or not even trying, I would personally take the genome with errors. Also, medical action is just one possible use (even then, the data should usually be OK): I think questions about ancestry and being able to read about on-going genomic research with your own genome in mind are also useful.

ADD REPLY
1
Entering edit mode

The key thing in what you've said is that the error rate SHOULD be <0.01%. Depending upon SOPs, your facilities and a myriad of other things, your error rate will vary.Having evidence of analytical validity would allow one to quantify that error rate explicitly, tested under different conditions, and put it on display for the public to evaluate as part of their decision making process. Analytical validity, specifically published and peer reviewed analytical validity, is a must have for any company wanting to promote a product that has any medical claim (I would argue this should be true for any PGS device so you know the results are meaningful- genomic data is noisy even under the best of circumstances) . 23andMe has CLIA, but I personally think you need published evidence; other companies have published their analytical validity results.

ADD REPLY
0
Entering edit mode

Yeah, I agree that a formal presentation of results on the company website will be useful. However, that <0.01% statistic comes from comparison of genetically identical 23andMe samples (either technical replicates or twins), so I think it is probably right.

I also haven't read the 23andMe peer-reviewed publications super carefully, so there may also be benchmarks in the supplemental data for those papers.

ADD REPLY
3
Entering edit mode
ADD COMMENT
2
Entering edit mode
11.1 years ago

Also, I think David Dobbs provides a pretty good overview of discussions from various sources on the web:

http://daviddobbs.net/smoothpebbles/feds-muzzle-23andme-decide-public-best-be-ignorant-about-genetics/

ADD COMMENT
3
Entering edit mode

IMHO the majority of published reactions seem to miss what the FDA's major issue is: about marketing and describing the service:

Compare the two:

  1. get your genome information from us - that is one thing
  2. diagnose your hidden diseases with us - that is an entirely different thing

It is about the combination and weight of these terms as used in the marketing materials and advertising that the FDA seems to be against, not the actual product. All 23andMe need to do is change the WORDS in their marketing and ads.

ADD REPLY
0
Entering edit mode

Yeah, I agree that the wording in this warning focuses mostly on marketing.

However, the FDA has previously tried to shut down DTC genomics companies (including 23andMe) and the 3rd paragraph seems to mostly focus on the accuracy of the test. The carrier status report should really be OK for diagnosis (and I think many of the specific associations mentioned in that warning are also pretty well established). Now, there are some caveats to some results like deciding how to combine independent SNP risks and explaining the difference between a mutations that guarantee onset of a disease versus modulate risk (which may only have a modest impact on risk in many cases). I personally think 23andMe does a decent job of this already, but I'm sure there can always be room for improvement.

In other words, my understanding is that the problem was primarily with direct communication with the FDA regarding technical benchmarks that would justify marketing claims (and I think the FDA is supposed to provide permission based upon this data prior to advertising). This certainly relates to communication with customers, but I think delays in formal responses to the FDA from 23andMe were the primary problem.

ADD REPLY
2
Entering edit mode
11.1 years ago

There is now a class action lawsuit against 23 and Me http://gigaom.com/2013/12/02/23andme-hit-with-class-action-over-misleading-genetic-ads/

I am more convinced than before that everything in this has to do with bioinformatics: it is about the accuracy of the methods, the claims that are made and of course the occasional exaggerations.

ADD COMMENT
2
Entering edit mode

That is unfortunate.

It does seem to reflect a relatively common sentiment among customers who didn't really follow all the scientific details, which is something I was surprised to learn. However, I think it was also influenced by a lot of misleading or inaccurate information out there. For example, this is the one that angered me the most (because it came from someone who really should have known better):

http://liorpachter.wordpress.com/2013/11/30/23andme-genotypes-are-all-wrong/

I'm hoping this can be settled out of court (if that is still possible at this point). At least this provides a list of specific claims that I hope 23andMe will directly address to customers in an official statement - at least they can reference a plethora of 3rd party experts who can generally back them up. I certainly think they made badchoices with the timing of advertising and providing terse official responses, but I don't think that should be a $5 million mistake (especially for a service that I assume is being provided below cost)

Update: I also have put together a survey on this topic. If you can fill out and/or distribute the survey, I would appreciate it!

http://cdwscience.blogspot.com/2013/12/survey-of-opinions-on-23andme-fda.html

ADD REPLY
0
Entering edit mode

Without agreeing or disagreeing about legal issues, I would like to know what you think is technically wrong with Pachter's article.

ADD REPLY
1
Entering edit mode

I for one thought the title wasn't quite right. It is equating the chance of making a single error in any SNP call with the genotype being wrong. After all every measurement has errors, by this definition every measurement ever made is wrong.

ADD REPLY
1
Entering edit mode

I had the same response. Read the title, thought "wow!", read the rest of the post and thought "what?"

ADD REPLY
0
Entering edit mode

I think the title might be a bit sensational, but I disagree that it is wrong, given what an error means for users of 23andme. From the article: "The fact that 23andme genotypes are wrong (i.e. at least one error in some SNP) wouldn’t matter if one was only interested in a single SNP. With very high probability, it would be some other SNPs that are the wrong ones. But the way people use 23andme is not to look at a single SNP of interest, but rather to scan the results from all SNPs to find out whether there is some genetic variant with large (negative) effect. The good news is that there isn’t much information available for the majority of the 1 million SNPs being tested. But there are, nevertheless, lots of SNPs (thousands) to look at. Whereas a comprehensive exam at a doctor’s office might currently constitute a handful of tests– a dozen or a few dozen at most– a 23andme test assessing thousands of SNPs and hundreds of diseases/traits constitutes more diagnostic tests on an individual at one time than have previously been performed in a lifetime."

ADD REPLY
0
Entering edit mode

Same as the others in comments and Biostar..."all wrong" is not going to typically be interpreted as "have some errors", especially for non-scientists who probably aren't going to read/follow the rest of the article. I agree most individual facts are accurate, but the way they are stitched together seems misleading to me.

ADD REPLY
2
Entering edit mode
11.1 years ago
Mary 11k

My favorite pieces on this were from

Laura Hercher: THE FDA CALLS A PENALTY ON 23andMe. I liked this one because it had a constructive piece at the end--let's build public tools then.

There is an opportunity here for the government and the genetics community to create a trusted source of information that is neutral, unbiased and supports a best-case scenario use of genetic testing by those eager to take the plunge. Hell, you could imagine tying in such a resource to something like ClinVar or GenVar, so that early adapters could contribute to publically accessible databases rather than giving it to 23andMe to sell.

and

Mike the Mad Biologist: Some (Hopefully) Final Thoughts on 23andMe. He doesn't dismiss the real consequences of misinformation like a lot of people I've seen have done, illustrating one example that a lot of guys aren't aware of.

ADD COMMENT
2
Entering edit mode
11.0 years ago
Carlos Borroto ★ 2.1k

New Blog post from 23andme.

23andMe Provides An Update Regarding FDA’s Review

It will be interesting to see if tools to analyze the raw data outside of 23andme start showing up around the web.

As we work with the FDA to get clearance, there will be some significant changes to the site. Customers who purchased kits on or after the FDA’s warning letter of November 22nd will not have access to health-related results. Those customers will have access to ancestry-related genetic information and their raw data without 23andMe’s interpretation of that data. They may receive health-related results in the future, depending on FDA marketing authorization.

ADD COMMENT
0
Entering edit mode

There are already at least two pretty good tools:

Interpretome: http://esquilax.stanford.edu/

Promethease: http://snpedia.com/index.php/Promethease

I also have some custom scripts that can be used, but I would probably rank the above options as better for most users:

http://cdwscience.blogspot.com/2012/06/my-23andme-results-getting-free-second.html

ADD REPLY
0
Entering edit mode

A comprehensive raw DNA data analysis site: https://www.dnatoz.com

ADD REPLY

Login before adding your answer.

Traffic: 2226 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6