NNDSS Evaluation Workgroup

Welcome to the NMI Evaluation Workgroup Basecamp! We will use this site as a repository for meeting and workgroup documents, and also to share ideas about evaluating and identifying improvements in processes related to MMG implementation, MMG onboarding, MMG maintenance, and Technical Assistance (TA).

📢 Untitled

Hello everyone,
Please let me know if anyone would like to share information about issues/ accomplishments related to MMG implementation/ onboarding in your jurisdiction, or any other topics during our next workgroup call (September 24, 3-4 ET).
Thanks!
Shaily

Comments & Events

Misty Johnson, Informatician, WI DHS
Yes, I'm happy to share our experience. WI completed a pilot of Babesiosis and Trichinellosis over the summer.
Shaily Krishan, Program Manager at CSTE
Thanks, Misty! I will add your topic to the agenda.
Rachelle Boulton
I would be happy to talk about our experience over the past year, as well. 
Shaily Krishan, Program Manager at CSTE
Thank you, Rachelle!
Nancy Barrett, Epi 4/PH Informatics Specialist
So our recent experience (slowly getting to success) is with FDD (we are in the MTB pilot as well, but I don't think we have anything unique to add there).

We have an issue that I'm not sure is for this group. We are working on implementing the FDD MMG. CDC added an option "Maybe" to some STEC interview questions, so the responses are Yes, Maybe, No, Don't Know. To add "Maybe" to the question list in our system is hours of work for an answer that really doesn't make sense (if you don't know, you don't know). So we are going to instruct interviewers to answer this as Don't Know and we are not going to add Maybe to our system (Maven).

My issue is really 2 things: 1) adding a response that doesn't make analytical sense or won't really collect meaningful data, and, 2) the "scope creep" that these MMGs have brought to data elements. Just because we are making the reporting electronic doesn't mean that we should be sending everything and the kitchen sink (yes, I'm older). And my gut feeling is that states are now expected to collect information that may not or is not useful at the state level -- which is where we need to keep our PH practice focused. 

Issue #3 is when we bring this up on the onboarding calls, our concerns are basically ignored. The CDC program in this case has already decided what should happen and is not listening or willing to consider that this type of change may not be epidemiologically useful or feasible to implement. And we are an original EIP state so we have tons of experience with this work.

Is this something for Michelle Hoover and EDX?

thanks for letting me vent!
Nancy
Shaily Krishan, Program Manager at CSTE
Thanks, Nancy. I agree about the "scope creep" issue, and we could perhaps discuss the workgroup's thoughts on specific information about the issues with MMG onboarding that we can pass on to CSTE leadership?

(Also, I think that sending this to Michele Hoover & EDX will be helpful)
Misty Johnson, Informatician, WI DHS
Nancy, as it's been explained to me by our enteric epis, there are many times in a food history that "maybe" is the best answer. Was there a bit of lettuce in your fast food hamburger? maybe. You can't really answer yes or no. The use of unknown implies the question wasn't asked or they wouldn't answer. I agree with you, and as a data person, I think in black and white, yes or no.  : )

We too have had experiences with vocabularies where the epis did NOT want to use an answer, so we just didn't add it to our vocab for that field.

We are only at the stage of modifying our forms to accommodate the FDD, so kudos to you for how far you are!
Teresa Jue
Hi everyone!

We've been experiencing some issues during our pilot for Measles/Rubella/Congenital Rubella, mostly surrounding the priority 1 and 2 data elements. Like Nancy mentioned before, it seems that the CDC programs have already decided what elements that would be required and is not as willing as we would have hoped to update some of this information. When asked for why the data element is needed, the explanation we receive back is that it is a priority 1 or 2 data element. Often times, we feel that the data element adds undue burden to our investigators, especially since the data may not be readily available. Now, as part of the onboarding process, there is an added step to send justification for why we are not collecting the data element(s) plus an added statement that we will consider adding it in the future if funding/resources allow for the enhancement. Not sure if this is going to be a required step for production on-boarding moving forward? 

It was my understanding that the pilot would allow for the requirements surrounding the specific MMGs and to better understand what jurisdictions are able to currently send, but I'm not too sure of that anymore. There is also inconsistencies with piloting and working with different program areas. Other programs areas we've worked with in the past have been more flexible about the data elements they receive.

Last thing, also related to piloting, is that there is an expectation for systems to change in order to align with the MMG. Which often takes more time than what the pilot timelines allow for (so basically, we're always behind the target dates.) 

To add on to Misty's point, we also have opted not to add all of the value set options if our epi SMEs review and decide that they don't quite fit what they're trying to look at.
Rachelle Boulton
I'd echo Nancy and Teresa's comments. We've been pretty disappointed with CDC's unwillingness to collaborate or compromise on data variables. We feel we (at the state and local level) have a perspective being closer to the data and cases, that would provide value to national surveillance, and it is for the most part discounted. It seems like the states first see the guides, for the most part, when they are first published for review on the webpage. It seems like at that point, CDC has put enough work into them, that they really don't want to make any significant changes, and especially when the requests come further down the line like in piloting. I'd love to see a more collaborative approach where CDC communicates with the states the type of information they would like to collect and the analyses they would like to perform, and we can work together to draft the questions in the best way, looking at existing guides to ensure harmonization, and collaborate on prioritization. 
Nancy Barrett, Epi 4/PH Informatics Specialist
Hi Shailey,
Time for another CDC "Listening" call? 
I think we (states) need to rely on federal funding because states chronically under fund public health at the state level and this puts states in a philosophical bind -- we feel that we must collect all information the funder (CDC) requests, even if the request does not make a lot of sense. So it's difficult to say no or question because no one wants to jeopardize their funding! And some states may feel they are in a better position/mind set to just ignore/refuse than others.

However...with scarce resources we need to be very cognizant of the impact and costs on states. As I see it, the real goals here are to: 1) improve integrated surveillance systems at states to collect information including follow up information in a centrally maintainable system, and 2) replace the reporting to CDC by paper/file uploads/Access databases, etc. with a single reporting stream. Too much scope creep during this process hinders reaching these goals. 
Molly Crockett
Agreed that a listening call is a good idea. I get the feeling that the EDX folks kind of get it, but that the programs are just in a constant state of raising and redefining expectations. I don’t remember who was on from CDC last time, but a presence from the programs that are driving the scope creep would be nice – to hear the message from multiple states at once.
Rachelle Boulton
We (Utah) had significant challenges seeing any progress with mumps, pertussis, and varicella for exactly these issues. We were talking in circles about the variables we would send, the value sets we were using, etc. I ended up having a conversation with Sam Posner, the Associate Director for Science at NCIRD, and when he got involved we were finally able to move forward. I think those are the people we need to engage, at a level higher than the CDC surveillance staff. EDX does understand our challenges, and they are sympathetic, but they really do have their hands tied. And the requirement to provide justification for why we aren't collecting it, and the promise to add it in the future just doesn't feel collaborative. 

I have had many discussions with the EDX folks about the CDC surveillance staff expectations, and they have said they communicate with them about the scope of the project, but I'm not sure how that is communicated or what exactly is said. I'd be interested to know if they have any formal documentation for surveillance staff that lays out the objectives of the project (what is in scope and what isn't), and a structured protocol/process for making this happen that includes when and how CSTE and the states should be engaged. 
Nancy Barrett, Epi 4/PH Informatics Specialist
Good points Rachelle!

Shailey -- I think we have our request. I'm thinking that those of us in this workgroup could put together slides for a call as well. We are definitely in policy and practice territory it seems.
Shaily Krishan, Program Manager at CSTE
Yes, we'll also discuss this internally with CSTE SI leadership to identify effective channels of communication with appropriate levels of CDC leadership. Let me see if we can schedule a listening call with CDC during one of the upcoming WG meetings, with appropriate program participation- We definitely need participation from Foodborne and NCIRD programs- what about STD/CS or MTB? 
 
Rachelle Boulton
Ideally, I think we should have representation from any group at CDC that has or will have a MMG. 
Shaily Krishan turned on public link sharing for this message on
Shaily Krishan turned off public link sharing for this message on
Shaily Krishan, Program Manager at CSTE
Hi everyone & Happy Monday!

We shared the CDC program/ MMG onboarding related issues with Kate Goodin, and will provide an update during tomorrow's workgroup meeting.

Misty Johnson, Informatician, WI DHS Misty - I will add your update about the WI Babesiosis and Trichinellosis pilot to the agenda.

Rachelle Boulton Rachelle and I will provide an update on our conversation with Kate

Nancy Barrett, Epi 4/PH Informatics Specialist Nancy Teresa Jue Teresa Molly Crockett Molly  we will also discuss the issues you mentioned and the next CDC MMG listening Session.

I will send the agenda out shortly- thanks! Shaily