CSTE National ELR Workgroup

Welcome to the CSTE National ELR Workgroup Basecamp! We hope you will utilize the space to share resources with colleagues, start discussion threads, and identify topics for the workgroup to tackle on future calls. All workgroup members will have the opportunity to join. Please note that you may edit your notification preferences to suit your needs (and your email inbox). If you are not already on the ELR distribution list and would like to receive call communications, please enroll using this subscription form (copy and paste in browser): https://app.smartsheet.com/b/form/5a91a4579952496682084796b82112e6

Issues with Test Volume

Hi everyone!

Rhapsody is having issues managing the unprecedented ELR volume we have been experiencing, and we are keenly aware that this volume is only going to increase. 

I'm wondering if anyone is having similar issues and have found ways of improving things. Our issues are summarized below:

1. webservice call is a major bottleneck.  Is anyone else using the Melissa personator webservice for geocaching address information?  If so, is anyone else having that be a massive bottleneck while processing messages?  At times, and usually these are midday and afternoon, we're struggling to get 10 messages to successfully process in a minute.  We've upped our Melissa comm point connections from 1 to 3 but that only got us up to 18 messages during these peak trouble times.  

2.  Large batch files.  We're having issues with large batch files from Labcorp causing massive amounts of our working hard disk space to be eaten up during processing.  We de-batch early in our route and process the individual messages through the rest of the route. It seems each route panel and each filter that processes a message will hold onto the full original debatched file.  With smaller batches this isn't such a big deal, but when a file has 1200 messages, and is 2.4mb in size, that immediately is 2.88Gb after debatching since the full file is sent with each debatched message.  Then we have 5 panels that have about 5 filters that do some processing on the message.  That brings us to 10 x 2.88 for 29 gb for one file processed.

We've requested that Labcorp split our files into Non-Covid, Covid PCR, and Covid Serologies so hopefully that helps with our file size and speeds the process up a little bit.  I suspect some of the file size problems might be contributing to some of the webservice call problems.

I'm guessing we're not alone in this and there probably aren't any good solutions, but hopefully someone is using Melissa and has found a way to improve that bottleneck, and maybe there's something we're missing in reducing the filespace required for larger messages.

Thank you, everyone!

Comments & Events

John Satre, Informatician at Iowa Department of Public Health
We came across the same two issues.

1. Increased Web service comm. point connections to 5. This improved the processing speed of the messages.

2. For Labcorp, changed the de-batching logic from filter to a new Directory comm. point with built-in de-batch functionality. This improved the performance to a large extent.
Mark Dittman
Andrew,

For the 2nd issue, we had a similar problem in PA with LabCorp when we moved to PhinMS and did not de-batch early enough in the process. This led to the exact same hard disk space issue. Try to de-batch as soon as possible and that will solve most of the drive issue. You may have to rewrite or reorganize your route workflows but you have little choice to reduce file sizes. Rhapsody is writing the entire batch to your message store for every single component it passes thru. We saw similar file size jumps and even higher in some cases.
Randi Hathaway
Thanks for posting this topic. 

In TN, we have had a lot of issues with processing large files in Rhapsody including files from LabCorp. Our Rhapsody programmer is currently breaking up large files prior to processing which sounds like what you all have already mentioned. This seems to have fixed the issue. 
Misty Johnson turned on public link sharing for this message on