MSAmanda, Sequest, Percolator discussion part 2

This is a continuation of a previous analysis from last week.  Part 1 is here.

Okay, here is the question.  What, if anything, is MSAmanda giving us that we aren't getting from Sequest + Percolator?  In the previous entry I think I did a good job of highlighting 2 things: 1) MSAmanda and Percolator work VERY well together and 2) We get more proteins from high resolution MS/MS spectra with MSAmanda.

I guess the question is this, at the peptide level, how many are unique?  Are we scoring the same stuff, mostly, or is this really complementary data.  In the end, does it really matter?  More peptides is a great thing, right?  But I want to have a solid metric (on one data set...) to say, "adding this search engine can give you XX% more results," or something.  There is lot of data out there regarding different complementary search engines used together, like this poster.  In general, however, I tend to expect an extra search engine to boost my IDs by ~10%.

First of all:  I mentioned last time that in this particular run, I was not happy with the Sequest + Percolator PSMs that made it through my filter.  I need to narrow those down to peptides that I trust.

I used the method that I mentioned last time:  I cut back my FDR cutoff at the "high" confidence level (since this is Percolator, this is based on q-value), until I got to consistently good peptides.

Here is a summary:
At 0.01, I had 11600 peptides from Sequest + Percolator
At q value 0.009, I had 11482, and they still didn't meet my threshold cutoff
At q value 0.006, I had 11,108, but I still didn't trust all of the lowest scoring peptides, and so on.
I ended up cutting it to 0.001, which left me with 10,226 peptides, ~800 less than I started with, but still a big boost over what I got from Sequest + Target decoy.  In a related note, I did this a second way, by cutting the original Xcorr factor to a minimum of 1.75 using the same sampling technique I liked the peptides and came up with close to the same numbers.  Interesting, but maybe coincidental.

Here are the peptide numbers from each analysis:
Sequest + Percolator:  10,226 (trusted)
MSAmanda + target decoy:  9262
MSAmanda + percolator:  12,241

And here is what it looks like:

Not a bad chart, right?  By the way, I'm completely fascinated by the fact that target decoy search sometimes gives me peptides that I don't get from Percolator.  It makes me wonder if we should be doing both in order to boost our ID counts.  Remember from the last entry that my quick and lazy analysis said that the peptides from both Amanda runs seemed trustworthy.

Anyway, I guess I was looking for a hard number.  So, if we add those up, it looks like we get 12,532 unique peptides from this one run.  And if we just look at the unique ones from Amanda + Percolator, we get 2086 (1585+501).  That is a 16% boost in trustable (that isn't a word either?  WV public education...) unique peptide IDs.  It's actually a little better than that since the total I have here also has the additional peptides from the MSAmanda + TD search, but I'm not going to do that math.  It's late, and this is reasonably close.

Okay, so I know running MSAmanda takes extra time.  But so does adding extra time to your gradient.  This is a 2 hour run that I'm analyzing.  If we added an extra hour to it we might have boosted our peptide IDs by another 10-20% (just guessing, but I should do that analysis, I have data just like that on a hard drive somewhere).  We could also have boosted this by running this same sample on a faster instrument like a Fusion.  We could also run it with a longer gradient + DMSO on a Fusion and do this, and you know what, we'd get a ridiculous number of peptides IDs. The point is, this is free data right there in your RAW file, you just need to take the time to pull it out.

TL/DR:  Are the peptides from MSAmanda unique?  A lot of them sure are!  When running vs Sequest in this dataset it gave us 16% new peptide IDs in exchange for a little extra processing time.

Fame in science

Uh oh!  This one has been percolating in my head for a while.  Do I write it and risk offending a lot of really smart people?  Or should I just do some vinyasa flow and just let this negative energy drain into the cosmos?  In the end, I did both.  My IT bands feel great, and I wrote something I feel is a lot more balanced than it could have been.

Let's start here:

In 2006, Anil Potti was a shining star.  He was in a fellowship at Duke and was on a streamlined path toward a full professorship.  Between 2006 and 2010, he published a slew of papers in all of the highest impact journals.  You see, Potti was a microarray expert while genomics was still the king of the roost.  During Potti's prestigious fellowship he had figured out how to decode the extremely complex relationships between drugs and cancer cell responses.  Figured it out?  He mastered it.  He could tell you from a microarray the target of the drug and whether it would work on one particular cancer cell and not another.  He got so good at it, in fact, that patients were being treated based off of what Potti could figure out about the microarray of their particular tumor.  The program was absolutely groundbreaking and signaled that genomics was finally coming into it's own and was going to change the battle against cancer into our favor.

There was just one problem.  The data was full of fabrications.  Microarray outputs are commonly converted into simple Excel format and you process from there.  I line mine up across, say control vs. treated, and divide to get my fold changes, sort by fold change and toss all the low numbers.  A number of short communications have been written about Potti's papers, like this one.  In them you'll find all sorts of fantastic observations, such as when the results didn't match what Potti wanted, he simply cut the columns and sorted them until they showed what he wanted.  He didn't do it a little.  He did it a lot.

In 2010, the papers began to be retracted, and the clinical trials were stopped.  Keep in mind, people were actually being treated with the chemotherapy agents that these fabricated microarrays were telling physicians to use.  The biggest problem?  The blatant errors in these microarray analyses were pointed out by a team at M.D. Anderson in 2007.  The primary author of the letter to Nature Medicine was Kevin Coombes, a guy who had written a whole bunch of proteomics papers and new a little something about reproducibility of -omics data.

And here is where, in my humble opinion, Scientific Fame came into play.  In 2007 the group at M.D.A., pointed out blatant errors in one of Potti's initial studies, and he immediately hit back. The M.D.A. evidence was solid,  but it was too late.  Potti's star was already rising.  And rising so fast that one detractor couldn't slow it down, and it wasn't until after it had gone to the worst possible level, to actually endangering the well being of real people in clinical trials.

And this is why I have a problem with this thing:

The Analytical Scientist (whatever that is) set up a ranking system based on a secret nomination and judging system to rank the 100 most influential scientists in analytical chemistry.  Are there some great scientists on that list?  Absolutely.  Are these people who have changed some of the fundamentals of chemistry and how we do it, possibly forever?  Yes there are.  I'm not arguing that there are some great people on this list.  What I'm arguing is:  whats the point?  I know the point for "The Analytical Scientist," this thing is generating some Ad revenue.  People like lists.

But here is the danger:  This is science.  We're supposed to be weighing out every idea by it's merits and by the strength of the proof behind it.  If we start to judge the idea by who said it, rather than purely by the merits of the evidence, then we've missed the point.  The next revolution in chemistry may come from a student at a small school with 300 students in the mountains of Kyrgyzstan and her ideas should be treated with the exact same degree of skepticism as the ideas of every person on this list.  I'm not saying that we're not doing that, but it sure seems like if we're going to invite the possibility of that kind of bias, then this would be a good start in that direction.

End rant.

Update:  Yes, I understand the hypocrisy in the fact that this is all being written by a guy who blogs a sizeable percentage of his thought into the universe every day.  That's what makes it fun(ny)!  Don't trust anything I write here, I try to warn you about my biases, and certainly don't think that I wouldn't have been super psyched if my name was on that list.  Maybe they'll extend it to the top 10,000 and I'll make the cut one day and I'll never write a bad thing about whatever that magazine was called again.

What is a TopN Peaks filter?

One at a time, I've been going through the PD nodes, new and old and evaluating them in exactly the way that one should.  Using my current favorite dataset, I simply add in the new node and run the same sample with and without this node.  It makes for some easy entries.  On the down-side, using just one dataset may not be an accurate representation of what this node can do, as they may be more useful for more specialized datasets.

The TopN filter is an interesting one.  It has two settings 1) the number of MS/MS fragments to look at and 2) the window width in which to look for these fragments.

For example, the defaults are Top 6 with 100 Da.  What this does is go through each and every MS/MS spectra and break it into 100Da windows.  Within each window, it determines the 6 most intense ions and eliminates everything else.  If you scanned from 400-1400, then you've reduced your MS/MS spectra to the 60 most abundant peaks and dropped a lot of noise from your spectra.

Sooooo... what does this button do!?!?
For one, it's fast.  On my current favorite dataset, a 2 hour HeLa high-high dataset, it takes about 2 minutes to run.  This is offset by the fact that the spectrum selector ends up taking less time.  My search using a Target Decoy ran 6 minutes, whether I used this filter or not.  Yes, my laptop knocks out PD searches in 6 minutes.  Let me know if you want the specs on it, it wasn't very expensive at all.

Okay, so there are no apparent consequences, time-wise, to doing it!  How are the peptides?
Well, in both the case of the target decoy and percolator searches, we ended up with slightly fewer peptides and protein groups when we use the TopN filter.  Yup, fewer.  End of entry.

Nope!  I'm joking.  Not about there being fewer peptides.  There are fewer, but remember a few entries back where I was talking about Percolator trying too hard on Sequest searches and letting some junk through?  What if there was now less of that junk?  That would be a perk, right?

And it is.  The number of peptides drops (from ~11,600 to ~11,100 in this search) but when you look at the worst scoring peptides that made it through the Percolator cutoff, they aren't nearly as bad.  The thing is that Sequest and Percolator are just digging too deep and making mis-assignments on what is essentially noise.  But if you do a good job of eliminating that noise, then we're looking at fewer false positives.

I encourage you to check out this node.  I'd love to know how it performs on a larger dataset.  I would expect it to work much better, but who knows.

OpenMS -- new algorithm for metabolomics

In press at MCP right now, is this paper:   "Automated Label-Free Quantification of Metabolites from LC-MS Data," by Erhan Kenar et al.,

Now, I know this is a proteomics blog, but I try to keep my ear to the ground in regard to this metabolomics thing that has been exploding.  And this is a nice new one.

First of all, it is built into the OpenMS platform, which has a big support network and is available on all platforms (and crazy easy to install!).  The cool part, however, is the use of a support vector machine (SVM) to rapidly and accurately identify metabolites.  A SVM is a supervised learning algorithm (think, Percolator, or artificial neural networks in genomics) that makes classifications in a non-probabilistic manner.  In this case, this sophisticated algorithm is used to determine whether ions in your run are metabolites of your ions of interest.

If you are doing metabolite ID and quan, you should take a minute to look through this paper and download OpenMS 1.11.  You can find it here.

Does our target threshold for MS/MS actually matter?

This very thoughtful question was recently posed to me by someone.  And I had a nice long think about it on a plane today.
Here it is:  If we are always going for our most intense ions, Top10 or 20 or whatever, would it even matter what we put our target threshold at?  Or would we never get down into that junk?

So here is my crude bumpy-airplane-ride attempt at an answer.
1)Start with my target dataset:  A 1ug Hela lysate ran on an Orbitrap Elite with a Top15 method in "high:high" mode with HCD (MS/MS at 15,000 resolution).

2) Filter for MS1 scans only
3) Set the bottom screen to only show the peak list
4) Go through all of the below analyses, return to Xcalibur, change the settings to "Display all" or you do a bunch of Excel work for nothing....erk....exactly why they serve alcohol on bumpy plane rides....
5) Export the peak lists.  For the sake of brevity, I exported one at each of these time points (in minutes):  10,20,30,40,50,60,70,80,90,100,110
6) Find out how many peaks are there and what the average intensities are

So, at the MS1 level in this extremely complex digest, each MS1 spectra contained, on average 2,413 peaks that Xcalibur could detect (+/-200 or so).  Of those peaks, the average recorded intensity was 1.2E5!

Okay, more maths:  How does this compare to our MS1/MS2 ratios:

Considering the length of the gradient, we ended up getting a full scan every 2.41 seconds in this experiment.

I'm going to need to make some big assumptions to do the rest of the math.  So bear with me.  I think it will be worth it.

Let's assume that we have a 30s peak width (should be close) and we'll assume that is uniform, so each compound is detectable in the system for 30s, and that is it.  This breaks our gradient into 240 measurable time windows, of which I sample 10.

Now, if we assume that the average number of ions around is a good measurement, 2413 x 240 gives us 579,000 ions that the instrument was able to detect and assign an intensity.  This is a tryptic run, but I'm going to ignore the fact that a lot of these are singly charged and unlikely to be sequenceable (which isn't a real word, I guess.)

So there are 579,000 ions and we looked at the most intense 26,494.  This is the top 4.2%.  Compare that to just the average intensity, and that means that at the absolute minimum, every ion we fragmented (given all of these assumptions are true) was at least 1.2E5.

This is on paper (partially on a napkin, to be perfectly honest), but according to the math.  No, there is no reason in a complex mixture to spend time fretting over whether you set your MS/MS triggering threshold to 5,000 or 2,000 or 500 or even 50.  If you're always going for the most intense ion, you'll never be digging into junk that low in intensity.

However, there are beginnings and ends to each gradient, and those shouldn't have peptides in them (in an ideal world they shouldn't have anything in them at all, but we all know this isn't how it works).  If your threshold is too low, then you will be triggering on noise there and increasing your file size, but that would be the only drawback.  From a statistical FDR, type level, this would actually be good for you if the premise that "the more bad MS/MS events we have for FDR, the better it works" is true (I've written about that somewhere in one of my previous and long FDR rants).

But, wait a minute!  Didn't I just write about the importance of thresholds in dynamic exclusion settings?  Yup!  I promise I'm going somewhere with this, but I'm out of time.  More later, maybe

TL/DR:  On paper (or on a napkin) there doesn't seem to be a good reason to worry about your minimum MS/MS intensity threshold cutoff in a TopN experiment in a complex mixture of fairly high load.

Dynamic exclusion -- which camp are you in?

I've noticed in my traveling that there are 2 very distinct ways that people set up their dynamic exclusion settings.  I've taken to referring to them as the two camps.  Here are some really crooked illustrations with somewhat arbitrary and not-to-scale numbers on crooked bars.
Camp #1:  The two-timers

The two timers will set their dynamic exclusion so that every prospective peak  has a possibility of being fragmented twice -- once at low threshold, then again near the peak apex.  This is done by using a low MS/MS triggering cutoff (think 2 E3) and then doing a repeat that approximates the half-peak width.  The illustration above demonstrates the ideal circumstance.  One MS/MS event at an intensity that may or may not give you a good fragmentation spectra, followed by a second that most certainly will.

Camp #2:  The soloists

The soloists give their instrument one shot to get a good MS/MS spectra for searching and then move on to the next target.  In general, I see the soloists using a higher target value than the two timers.  The benefits here can be big.  If you only fragment each ion once and it fragments successfully the first try, then you can fragment double the number of ions/run as a two timer using otherwise identical conditions.  The downside occurs when that one fragmentation event isn't enough to efficiently ID your peptides of interest.

Now, this is the end of this for me.  I won't tell you which camp I'm in, because I can too clearly see the benefits of both and I struggle with it every time I set up an experiment.  I just wanted to clearly identify the two groups so that we know what some people are doing and to set the backdrop for the experiments I'm currently planning to perform.

Do we still need to fractionate our samples for deep proteome coverage?

Here is a good question:  Are we still in a place, technologically, where we gain more from pre-fractionation than we lose?

At first, we had to, right?  If we didn't cut gel slices or fractionate a sample by SCX or IEF or some other manner all we'd ever see was albumin and keratin from Pan troglodytes.  Maybe that was just me.  My first several proteomics experiments and that's all I got.  Chimpanzee keratin, because mine is more similar, apparently, to chimp skin than human.  I have proof, btw, I'm not just being funny.

Fractionation seemed to be the key.  20 fractions yielded more than 10 fractions and we dealt with the fact that we went from a 24 hour run time to a 48 hour one, because we were getting some sample depth.

Here is the question, though, is it still necessary?  Or have we attained the speed and sensitivity in MS and the quality of nanospray chromatography separations gotten to the point that the inevitable losses incurred by pre-fractionation and the staggering increase in run time are far worse than the gains?

The literature this month would go in two completely different directions on this one.  Two weeks ago I summarized a paper in Proteomics that used a pretty unique 3D fractionation method and a lot of you guys really liked that paper.  Going in the complete opposite direction, the paper from Josh Coon's lab with the one hour proteome showed that you can get into the high numbers with just 1D, if you optimize the LC really well (oh, and have a Fusion).

Here is a new one for camp number 2:  "Rapid and Deep Human Proteome Analysis by Single-dimension shotgun proteomics" by Pimoradian et. al., out of Sweden, the home of the world's greatest band (In Flames), and also the home of Roman Zubarev's, who is corresponding author on this work.  From the title, you might be able to guess a little bit about the paper...but I'll still tell you more.  By using a 50cm column and optimizing the LC conditions to the ideal dynamic exclusion settings (see, I told you this was super important! see rant #1, rant #2) , they were able to knock out 4,800 unique proteins out of an A375 cell pellet.

I'm going to reserve my opinion for now.  Nope, I lied.  I think that if you're currently doing lots of prefractionation or 3D separations, I think that you owe it to yourself to step back away from your system and give one of these optimized 1D separations a shot.  The evidence is building.

You can link to the abstract for Pimoradian et. al.,  here, but the pre-print release link has expired.

TMT 10plex on the Q Exactive Plus!

Yes, I'm re-using this image.
Why, you ask?  Because I am looking at data that you can generate from using the TMT10plex on the Q Exactive Plus.
Imagine a world where your isolation mass width is ~1 Da with no loss in sensitivity.  Imagine that the reporter ions that you have on 10 channels for quan are resolved at 50,000 resolution at the reporter ion level (35,000 resolution setting).  Wait.  Don't imagine that part, here's a picture of a randomly selected peptide reporter pair at 1:2 ratio (honest to Gus, it is random, they all look this good!)

Then imagine that you set up a series of controls in a 1:2:4:8 ratio, and they all turned out that way, with a <15% variance from expected at the protein level.  As a side note, it might be interested to know that this data came from quan on ~4,000 protein IDs.  Yup!  I'm not making this up.

That has been what my last 2 days have been like.  Lots of time thinking about how many iTRAQ 8 plex samples I have ran over the course of my career, and how little interesting data has come from them.  The future is looking pretty bright.

By the way, the method that my friends and I generated on this amazing instrument is now uploaded to the Orbitrap methods database.  Yes, the QE Plus is more sensitive and can handle a more narrow isolation width, but widen the window a little and this is how you should run TMT10plex on your Q Exactive.

Now, my disclaimer.  Did the Ting et. al., MS3 paper show that narrowing the isolation window didn't help all that much with the reporter ion suppression?  Yes it did.  Ultimately the MS3 method is better.  But narrowing the isolation most certainly does not hurt!  All it can do is eliminate co-eluting species and improve your quan.  And if you fall into my camp of people who need to run faster with more sensitive identification at all costs, even that of reporter accuracy, then we take the improvements we can get without compromising speed.

Broad researchers reassess high throughput cancer screens with good results

Last week in Boston and I was informed that Broad rhymes with toad.  The way I was saying it was wrong, and in the absolute wrong context in casual conversation could be misconstrued into somewhat sexist statement.  So Broad, like toad, everybody!

For everyone else in the entire world who already knew that (my home state is infamous for saying things incorrectly, by the way, it's just how we roll), here is some really cool science that they are doing at the Broad Institute of MIT:

High throughput cancer drug screens normally work like this -- hundreds or thousands of plates, cells, or wells of immortalized cancer cells are grown under identical conditions.  An automated system doses each separate cell with a different prospective drug or dosage of said drug and the efficacy of the drug is recorded by means of the destruction of cells or the inhibition of cell division or something similar.

Researchers at the Broad took a step back and decided to make the system more physiologically relevant.  By mixing leukemia cells with normal stromal cells, they can more accurately mimic the microenvironment that these cells exist in.

What they found is that they can eliminate some false positives.  Some drugs will work on leukemia cells alone, but are protected by the presence of normal stromal cells.  Re-screening potential drugs with bring you to drugs that not only work in the well, but are also more likely to work under normal physiological conditions.

You can read more about it in this press release that I found through Twitter.

QE Plus, first impressions

I just wrapped up my first day on a QE Plus.  This was a full out shotgun proteomics instrument -- no extended resolution (280,000 resolution is an option) or protein mode (for improved intact analyses).  It is the "base model" plus.  First impressions; if you've ran the QE, you are ready to go on the Plus.  The controls have not changed noticeably, in fact, if you've purchased a QE Plus, you can still get going by using my QE training videos without much of a hangup.
An option that I like a lot:  you can set your mass tolerances for your inclusion, exclusion, and dynamic exclusion lists where the original QE is set for 10ppm (a perfectly suitable window for almost every applicaton!).

Aside from that, from the instrument setup side, it is every bit as intuitive and streamlined to calibrate and set up methods as I've come to expect from this great instrument.  All improvements in engineering and sensitivity have taken place inside the box.  We experimented with some very tight MS/MS isolation windows on overnight runs and I should have something to say about the power of the new quadrupole sometime soon.