FAQ - What is the difference between an observation and a nonconformity?

AASHTO re:source Q&A Podcast Transcript

Season 4, Episode 18: FAQ - What is the difference between an observation and a nonconformity?

Recorded: September 19, 2023

Released: September 26, 2023

Host: Kim Swanson, Communications Manager, AASHTO re:source 

Guests: Amy Reed, AASHTO Accreditation Program Manager, AASHTO re:source 

Note: Please reference AASHTO re:source and AASHTO Accreditation Program policies and procedures online for official guidance on this, and other topics. 

Transcription is auto-generated. 

[Theme music fades in.] 

00:00:02 ANNOUNCER: Welcome to AASHTO resource Q & A. We're taking time to discuss construction materials testing and inspection with people in the know.  From exploring testing problems and solutions to laboratory best practices and quality management, we're covering topics important to you. 

00:00:20 KIM: Welcome to AASHTO resource I'm Kim Swanson. We are without Brian Johnson today. But with me today is AASHTO Accreditation Program Manager, Amy Reed. Welcome back to the podcast, Amy.

00:00:32 AMY: Kim, thanks for having me again today.

00:00:34 KIM: So we have Amy on... we - I have Amy on today to talk about the differences between a nonconformity and an observation. I know we get this question a lot. It is a FAQ so I just thought Amy would be an excellent person to discuss this with us so. Let's just dive in Amy. On an assessment report laboratories are going to see observations and nonconformities. Why is there a distinction between the two?

00:01:06 AMY: We also have informational notes. I don't know if that's something you want me to go over today too.

00:01:11 KIM: Whoa, OK. Mind blown. Yes, I think we should go over all three of those things. Yeah. So what? Why are there 3 different types of notes mentioned in an assessment report.

00:01:23 AMY: So I'll start with informational because that one is pretty much the most basic from a laboratories perspective, it's the best you don't have to do anything. It's kind of a heads up. So we may include an informational note in your report if there's some changes coming to a standard like R18 or the app Procedures Manual doesn't require any immediate action right now, but it's something you should be aware of. So sometimes those are included. Informational notes don’t count against you in any way though.  Next would be observations, and those are things that were found to be a technical error or deficiency during your assessment. But we've determined that we think those are minor enough that you can correct those on your own before your next assessment. So, you still do have to take corrective action on them. You just don't have to upload it online to resolve the nonconformity. And the nonconformities, that's what everyone always is worried about. What is going to happen if I have a nonconformity? So, they do happen.  It's a great learning experience for your laboratory, but those are something that you need to address with your quality analyst. Once your final report is issued, you'll actually upload evidence that you resolved that nonconformity, and all of your nonconformities will have to be resolved before your accreditation continues satisfactorily.

00:02:35 KIM: I like to visualize things. What I just heard was that with the three different types of notes that will be on an assessment report, you can kind of think of them as like a green light, yellow light, red light sort of thing. So, the green light is the informational where it's just proceed as is. No worries. Where the yellow light would be the observation and that is more of a you have to do something. But it's not like major. And then with the red light would be the nonconformity. And that is definitely you have to stop and pay attention to what that note.  So, I guess let's dive into observations a little bit more.

00:03:24 AMY: Most things are nonconformity. There are not many things that are considered an observation, but it's going to be something that isn't going to affect a laboratory's ability to perform testing and get accurate results. Maybe a minor overall organizational failure, such as you missed an interval. So, your interval for checking your balances is 12 months and you went 18 months. They were current during your assessment, but you don't really have a way to go backwards and fix where you missed that interval in the past test. So we think as long as you have a corrective action, you keep on file yourself that says, hey, this is what went wrong. We're going to do this to make sure that interval isn't missed in the future.  That's probably satisfactory, not something you have to submit to your quality analyst since it was current during the assessment. But if that happens again during your next assessment, then it's a much bigger problem.

00:04:14 KIM: OK, so the issue is if you have an observation in one assessment and then your corrective action.  Either didn't happen or wasn't effective, so you have a repeat. The next assessor observes the same thing at the follow-up assessment. Then can it become a nonconformity or will it stay in observation?

00:04:37 AMY: Two observations become a nonconformity automatically and they don't have to be identical. So, say one of your technicians was missing the interval for retraining for soil standards this time and next time a different technicians interval was missed for aggregate standards, it's not the same technician. It's not the identical test, but it's the same problem. So, it's still going to count as a repeat. It's still going to turn into a nonconformity. Because the laboratory wasn't able to address the bigger issue of why these intervals are being missed.

00:05:06 KIM: That's, I think, a great point. I didn't even know that it can be just similar enough and it's the same root cause so to speak. Or the same systemic issue that then can turn it into a nonconformity. Whether when it once was an observation.

00:05:24 AMY: Yeah, it's really good to remember that. I know labs are happy when they get an observation, but you should take it just as seriously as a nonconformity. There's still action that needs to be done on your end and you really don't want a report that says, you know, this is the second time this was brought up to your laboratory and it still hasn't been resolved. So another thing to think about, and Kim, you've probably heard this before on podcast, but it's always great to look at your previous report before your assessment to make sure you have corrected everything and it's in place so you don't get those repeat nonconformities.

00:05:57 KIM: Yes, I think that is a good reminder and it has been a minute since I've heard that on the podcast that I have heard it so that I think that's a good reminder for our listeners to review your previous assessment reports to make sure that you don't repeat any of those or that your corrective actions were actually effective.

00:06:13 Amy: Absolutely.

00:06:15 KIM: So, when it comes to nonconformities, what are the basic types of nonconformities that a laboratory will see in their assessment report?

00:06:22 AMY: So we kind of categorize them into three main categories, and from there you know they branch out a little bit, but we're going to have equipment nonconformities, procedural, nonconformities and then also nonconformities related to quality management system policies or procedures.

00:06:37 KIM: And I do know from the podcast that each type of nonconformity requires submitting different documentation or different evidence of correction for your quality analysts. So for a procedural type nonconformity, what is a quality analyst looking for from the laboratory to show that that's been resolved?

00:06:58 AMY: So, this one is usually the most straightforward, and it's just a confirmation from the laboratory. They need to submit evidence that whoever performed that testing during the assessment has been retrained and knows the proper procedures to perform it. They've been observed performing it correctly. And that will hopefully not be an issue going forward for them.

00:07:16 KIM: All right, for in a nonconformity based around equipment. What type of documentation or corrective action kind of things are the quality analyst looking for from a laboratory?

00:07:27 AMY: So, from a piece of equipment just needs to be fixed and you're not purchasing new equipment, you would submit some kind of explanation of how it was fixed, if any pieces or parts were purchased or if you had to do a work order and someone came in and fixed it. You would submit that for as a reminder, anytime equipment is damaged and it's put back into service, it does need to be recalibrated or standardized or checked. Again, if the laboratory the equipment's at the point where it's broken and new equipment needs to be ordered, we do require an updated inventory list that includes that new equipment we require a packing slip for the equipment. Knowing that has actually arrived in the laboratory versus just an e-mail confirmation that it was ordered, because we all know things happen with shipping or orders get cancelled sometime. So, we require it to be in the laboratory and then also as I mentioned with equipment that's been moved out of service and put back into service after repair. There, if the equipment does require any kind of calibration, standardization or check that has to also be submitted to us for equipment nonconformities.

00:08:28 KIM: And then for the final kind of category, you said it was based on procedural and quality management system. What kind of resolutions are the quality analysts looking for those types of nonconformities?

00:08:44 AMY: Yeah. So, if it's like the laboratory has a policy and the policy is missing requirement. E329 for example one of the ASTM quality management system standard.  It's gets really specific into the requirements of the each laboratory’s confidentiality associated with that. And so that's a common area where we see nonconformities. To resolve that a laboratory would just submit a more detailed procedure explaining how their lab handles all of those individual requirements in E320. Say they do have to submit the actual updated policy to us, and that's because if not typically it turns into repeat nonconformity because someone will send us just something from a Word document or something they just typed in there saying, hey, this is their updated policy and then they forget to update it in the quality manual. So we need the actual page from the quality manual or the entire quality manual with the laboratory then saying hey, you can find this on page 27. And that's just to make sure it's actually been implemented into the quality manual and then if it is related to a document from the quality manual that's missing. We would require that document, for instance, if your internal audit was missing a review of low proficiency sample scores, we would require an updated internal audit that includes review of those along with everything that was on your previous annual review, too.

00:09:58 KIM: OK, now. Speaking of internal audits, I am kind of familiar with some of these terms just based on our internal audits that we had. So for laboratories that are doing their own internal audits, do they need to use the language of observation and nonconformity, or is that up to them absolutely up to each laboratory? 

00:10:25 AMY: We don't want to say this is your quality manual. This is your quality management system. This is what you have to say. Instead, R 18 and the AAP Procedures Manual gives you a guideline of what your laboratory should be doing and then from there you put in terminology documentation that works for you, because if we're just handing something as the accreditation program to you, it's likely it doesn't work for every laboratory, so it's really important that each lab customizes that to what works for them. 

00:10:45 KIM: That’s good to know. I think for laboratories that they don't have to use the language that we use as their third-party auditor for their internal things. So, we talked about earlier of when an observation can kind of become a nonconformity if it's been repeated.  What happens if you have a repeat nonconformity from assessment to assessment.

00:11:07 AMY: Yes, it is something that all of her staff had asked her resource. When they're out doing assessments, they do compare the previous report to the current report. And as I mentioned, the nonconformity may not be identical, but if it's systematically the same problem, it's going to be upgraded to a repeat nonconformity and we require a little bit more in those cases because our hope is that you get it resolved after that first assessment and if it's not, we want you to dig deeper to figure out what's going on. So, we require a corrective action report every single time for repeat nonconformity or for an observation that's been upgraded to a nonconformity and in that you need to, as a laboratory dig pretty deep and figure out what happened. Why was whatever you put in place last time, not enough to prevent it from happening again.

00:11:50 KIM: Since there's kind of, you know, stages of a nonconformity, you have the regular nonconformity. Repeat nonconformity. Is there a...three-peat nonconformity. I don't know what the third, repeat the third time would be called. Yeah, but is there is there like... What if it doesn't work and you have it on 3 assessment reports? What happens then?

00:12:09 AMY: Unfortunately, it does happen. Not a lot. I will say it doesn't happen a lot, thank goodness. But yeah, sometimes something's on there three times. And that's when, as a quality analyst, they're going to keep going back and forth with you and say.  You know, did you get to the bottom of this? Is this really what happened or it's like a procedural nonconformity and your texts keep missing, you know, multiple parts of the same test. We may suspend your accreditation until you have a supplemental assessment. We may require extra proficiency samples be sent to your laboratory and be ran. It just depends on what the nonconformity was. What you're saying, your corrective action is, and some of that is customized to each laboratory and they'll, you know, they'll need to resolve it.

00:12:46 KIM: OK. That makes sense. Now we've been talking about AASHTO Resource assessment reports. Do CCL reports look differently or of different categories of findings?

00:12:59 AMY: Yeah. So their format is different, which can be really confusing when you're trying to juggle between an AASHTO resource assessment and assess your assessment. Sometimes they happen close together and you're like, wait, this is a different format. What's going on. I will say it's the same quality analysts, the same group of QA's that are reviewing reports for CCRL and AASHTO Resource, but the report structures different. That the requirements are going the same. CCRL just calls them footnotes instead of nonconformities. So if you ever hear footnotes, it's the same thing as a nonconformity. CCRL does not do a comparison from one report to the previous report to look for repeat nonconformities. That's something that we sometimes catch on our end. We don't always though unfortunately, but we really. Hopefully leave that up to the responsibility of the laboratory that if you know it was a problem during your previous assessment and it happens again, that you dig a little bit deeper. But when we're comparing reports and we see that we do let the lab know, hey, this was also an issue during your last seizure report. I'm going to need. That corrective action report explaining what happened.

00:13:59 KIM: All right and. Is there the equivalent of an observation or informational, or is it just the footnotes equal nonconformities?

00:14:08 AMY: Just footnotes is what they note in their reports. 

00:14:10 KIM: Yeah, so. OK, CCRL - and I may take for granted that people don't know that that's the Cement and Concrete Reference Laboratory, but CCRL their footnotes, are they broken up into the same three basic categories-Ish of procedural equipment and....?

00:14:31 AMY: Policies and procedures from the quality management system standards.

00:14:35 KIM: Yeah, like I just said procedure, how is that different? But it's it is different.

00:14:39 AMY: Equipment, procedures and procedures for training your staff, those sort of things, so you could just say policies if you want. So physical demonstration is the other kind of procedure so. We call them procedural nonconformities, but it would be the physical demonstration of the test during the assessment.

00:14:56 KIM: The procedure of the demonstration versus the policy of a procedure. Yeah. Now that I’m  in thinking about it, the more complicated it's in my head. But let me ask that question from the beginning. So do the footnotes for CCRL reports kind of follow the same, I'm assuming the answer is yes, but the same kind of breakout of procedural policy equipment, or are there different ones for CCRL?

00:15:27 AMY: They do so even though the reports are categorized, they look a lot different than resource reports. It's the same process. They still have inspectors that go out and perform the assessment and they still review the quality documents and they still issue a report. Still looking at the same kinds of standards, just for different scopes. They still review R18 standards, they as team quality management system standards. The nonconformities or footnotes might be worded a little bit differently, but this same exact corrective actions will have to be performed by each laboratory to resolve them.

00:15:56 KIM: That makes sense, and you've mentioned that laboratories for repeat observations or repeat nonconformities will need to fill out a corrective action report form. Is that a standard form or is that any form that works for them that they've created at their laboratory or is it in the accreditation then system what does that look like?

00:16:19 AMY: So we have an optional form that laboratories can choose. If you go to AASHTO Resources website and I don't know can you might be able to link it here even [KIM: I will.] but in re:university we have corrective action reports for assessments. And that's what you could use to. About, it's totally optional. Some laboratories have a great system internally that they use instead and will always accept something different as long as it works for your laboratory. We just sometimes provide resources for those labs that want a little head start. They don't know where to start with the forms or the. Templates so we do have some available.

00:16:50 KIM: Right and if laboratories are using their own internal forms, what things should be on that to make it effective for them?

00:17:00 AMY: Yeah. So they need to start by indicating. What went wrong? So that's typically just what the nonconformity is and then after that it's going to be what works. Some labs do like 5 Whys is like. Why did this happen and they just keep digging deeper some.  Labs just do a couple interviews around the lab. Whatever works for them, they can do, but you're going to document exactly what you did at your laboratory. So you write down what the nonconformity was, how you investigated it, to determine what the problem was and then. Once you've investigated it, what steps you've put in place to prevent it from happening again and fixing anything that might have, you know, gone wrong this time because you missed it. For instance, if you missed reviewing and I brought this up earlier but low proficiency sample scores on your internal audit.  And that's repeat nonconformity. Well, you're going to be like, well, what happened? You know, we had it last time it we resolved it and it's like oh, we forgot to add. It to our template. So then you make sure that all the templates everyone are is using on your staff as the new template and then you also have to then use that new internal audit template to conduct an internal audit. And submit to your quality analyst and internal audit that does cover every section that's required.

00:18:11 KIM: All right, so once a laboratory has their assessment or inspection report. And they see their nonconformities observations and all that fun stuff and informational notes. How long do laboratories have to get this problem resolved?

00:18:28 AMY: Yeah, so this process is the same for an AASHTO resource assessment or CCRL assessment because all of that is filtered through the accreditation program, but once the final report is issue, the laboratory will their primary and secondary contact will receive an e-mail notification letting them know that their final reports been issued. They can go online and start resolving nonconformities and uploading.  And it also gives them a 60 day deadlin60-dayew years ago that used to be 90 days and we found that most of our customers waited till around the 88 to 90 to start submitting. And really what happened during your assessment isn't fresh in your mind at that point. It's important that you start that process sooner to figure out what happened. So that's why we moved it to 60 days.  You know the better though, after that final report is issued, you want to start submitting responses to what happened and getting a clear plan of action and letting us know if you know something will take beyond sixty days. The rationale for why.

00:19:21 KIM: And so that's 60 days from when the final report is released, but laboratories get an unofficial report at the end of the assessment is that that's correct, right?

00:19:33 AMY: That is correct and that time can vary. So laboratories regardless of who their assessment was through, they'll get that unofficial preliminary report after their assessment and then they'll receive the notification once the final reports issued. If one of the organizations is really on top of getting files processed or it's a rush, it might be within like a week. The final reports issued. Sometimes it does take a few weeks though, especially from central reports because they have to go through the review process on CTRL side and then they're emailed over to Astro Resource and then we take care of uploading them to our system.

00:20:04 KIM: You could have bonus time, so to speak, from when the preliminary report is issued to when the final report is issued. Would that's not free time. You should still be looking at and starting to investigate that, because I can't imagine that much is going to change from the preliminary report to the final report.

00:20:24 AMY: You're correct. So absolutely. As soon as that assessor walks out your door is when you should start thinking about how you're going to respond to these nonconformities. Because that's when you just had those discussions. You know what went wrong. And it's hard to remember after a few months because you have a lot of other stuff going on, be on your assessment with us and that preliminary report is almost always identical, except for editorial changes. So assessors are kind of rushing when they write those, so there may be some typos here or there that need to change, or the standard wording might have to be updated a little bit. But in general, the content of the report does not change if it's going to change, the assessor will reach out and let you know what happened and why we're going to change something.

00:21:02 KIM: And I do remember we had a podcast episode few seasons ago, I think, and this what you were talking, just made me think of it is that we no longer allow for laboratories to resolve findings or nonconformities at the time of the assessment.

00:21:22 AMY: That is correct. In the past, they were called. RDA's – Resolved During Assessments. And in some cases, assessors were given direction that they could resolve something, such as like a policy if it was missing a requirement for R18 or E329, it could be added and as long as it was shown to the assessor while they were still there at the laboratory, it would be marked as resolved during the assessment. That it is still noted in the report in the final report, it was just the laboratory didn't have to respond to the quality analyst, but we did away with those a few years ago and the rationale behind that is.  We were seeing a lot of repeat nonconformities for these, and that's because you're trying to rush, get that policy updated, show it to the assessor, the assessor says. Yeah, that looks great. And then you jump back on doing your everyday work. And because it wasn't something you had to respond to, you forget to upload it where it needs to go. You forget to put it into practice. So that was one of the reasons that we stopped doing the RA's. The other is that that deadline is kind of Gray for some people. And they're like, oh, well, as long as I can get it.  To the assessor before the final reports.  Issued. They won't put it on there. It'll be fine. And then it got really stressful for some of our staff because they were already working with another laboratory.  And the previous lab was reaching out with questions and saying, is this acceptable? What if I change this and it just it got a little bit overwhelming and on the quality?  Analyst side, we are. Used to doing that back and forth, but it's really difficult for staff that are out in the field.

00:22:45 KIM: I can imagine. I would definitely not want to be the assessor having to deal with all of that as well, but I think your point to not really resolving the issue is the main point of not doing that. So that definitely makes sense to me. Now, did I miss anything? Amy, is there anything I should have asked that I didn't?

00:23:06 AMY: Think as always, if you feel like you're not getting a response fast enough from your quality analyst, please reach out to your quality analysts. Sometimes they have, you know, a lot of labs are working with, so reach out and let them know that you know you've submitted. This is what's the next step for you and they can let you know if there's something else you need to do in the meantime. Or, you know, they'll say, OK, you're here. You are on our list. This is what we need to do. Or if you don't hear back from the quality analyst and you're curious on where your lab is in the review process, you can reach out to me and I can look into it for you too.

00:23:36 KIM:. That is a great reminder, Amy. So thank you very much for being our guest today and going through the nuances, it was definitely more nuanced than they thought, and from the differences between observation and nonconformity. So thank you for being with us today.

00:23:54 AMY: Thank you, Kim. It was great to be on the podcast again.

[Theme music fades in.]   

00:00:00 ANNOUNCER: Thanks for listening to AASHTO re: source Q & A. If you'd like to be a guest or just submit a question, send us an email at podcast@aashtoresource.org or call Brian at 240-436-4820. For other news and related content, check out AASHTO re:source's social media accounts or go to aashtoresource.org.