Tuesday, May 2, 2023

Tips to speed up SpectroNaut runs without compromising data!

Disclaimers over there --> this just appeared to work for us. The same big batch of files that were taking forever finished overnight after I made these changes. Disclaimers --> 

This is going to be very quick and largely unscientific, but if you're facing huge SpectroNaut bottlenecks like we have been, all of this has helped. 

1) Run everything to and from a high speed solid state drive (especially the Temp folder)


This might have been one of our biggest bottlenecks. We have this huge NAS drives for data backup on our PCs. We know for sure that processing FragPipe off these drives takes a lot longer than off of our primary solid state drives. I only found this when one of our NAS (network assisted storage?) drives was reporting that it was almost full during a 128 file processing queue.

According to this computer nerd page, our NAS drives read/write at less than 200 MB/s


Our newest processing SSD (according to the vendor) has a R/W speed of 7,000 MB/S


35x faster? 

2) We put in more (and faster RAM) 

Our PC had 64GB of RAM that was rated at 2.7 whatever units. For $700 we were able to take that out and put in 128GB of RAM that was rated at 3.7 whatever units. 1.6x faster? Maybe? 

3) Turned up the fans on our processing PC and moved the settings from "Quiet" to "High Performance" 

This box has an AMD processor and I found some software from the vendor that allowed me to see what was going on with all the PC things. There were settings on the water cooler, etc., where I could choose "quiet", "balanced", or "high performance". Obviously I put it on the last one. AND it allowed me to change the lights on the PC. Since we know that PCs need to be cooler, I set all the lights inside the PC to blue. The PC is now louder and, at least at a psychological level, it appears to be much faster.

4) On big queues >100 files, I've also found that converting the data to HTRMS first appears to help. There is a little utility with SpectroNaut that can do this, however, if your 100 files are 400GB, you may now have 800GB of data, but these HTRMs things do appear to process faster (thanks Dr. Neely for this tip). 

1 comment:

  1. What’s your experience / advice regarding centroided data vs profile data? I heard that most algorithms convert profile data to centroided data anyways. One might save a lot of space and processing time by directly recording centroided data! I remember an old poster from spectronaut about that ...

    ReplyDelete