The Value of Internal Audits - Part 2

AASHTO re:source Q&A Podcast Transcript

Season 4, Episode 25: The Value of Internal Audits - Part 2

Recorded: October 25, 2023

Released: November 14, 2023

Hosts: Brian Jonson, AASHTO Accreditation Program Director; Kim Swanson, Communications Manager, AASHTO re:source 

Guests: Tracy Barnhart, Quality Manager, AASHTO re:source

Note: Please reference AASHTO re:source and AASHTO Accreditation Program policies and procedures online for official guidance on this, and other topics. 

Transcription is auto-generated. 

[Theme music fades in.] 

00:00:02 ANNOUNCER: Welcome to AASHTO resource Q & A. We're taking time to discuss construction materials testing and inspection with people in the know.  From exploring testing problems and solutions to laboratory best practices and quality management, we're covering topics important to you. Now here’s our host, Brian Johnson.

00:00:22 BRIAN: Welcome to AASHTO.

00:00:22 BRIAN: Resource Q&A I'm Brian Johnson.

00:00:25 KIM: And I'm Kim Swanson and welcome to Part 2 of the Value of Internal Audits. We are joined again by AASHTO resource Quality Manager, Tracy Barnhart, thanks for being here, Tracy.

00:00:39 TRACY: Thank you. Good to be here.

00:00:40 KIM: Now, in last week's episode in. Part one we. Covered. Kind of. The basics of an internal audit and what to do and how to prepare. We talked about interviewing people and things like that, but I. Really in this. Episode want to deep dive into the value that we see in the specifically?  Brian of the internal audit process looking. At a recent AASHTO Accreditation Program, internal audit from this year and really deep dive into that.

00:01:13 TRACY: I would love. To hear Brian's perception of internal audits and and how they have helped you as. A director of a program.

00:01:21 BRIAN: Well, they're always useful. I get so much out of the internal audits and I enjoy them, even the.  So you go into an internal lot and you think, alright, what's going to happen this time? What kind of strange thing is somebody on my staff going to say to the internal auditor or misunderstanding? Is there going to be? But those are the good things. Those are where you learn and that's where you improve. So I invite those conversations, the talking part. I love that point too, because you know, most people do, and the internal audit after they've reviewed everything they write up the report, that's it. But that's the beginning for you. That's the prep part. So one thing I want to talk about with our internal audit is the way Tracy organizes these is very useful in the way she shows her work.

00:01:55 TRACY: Right.

00:02:05 BRIAN: Or prepares the record. She has a list of personnel interview during the audit, so she makes a plan like she mentioned and sets up interviews with different people. And sometimes she'll add people so like, as she's going through the audit, let's say she talked to one of our administrative assistants and said, hey, so how do you do this? And they say, oh, and then I. And then I talked to this person and Tracy's looking at the procedure and says I don't see. Down there. That's strange. I wonder why that person's involved. So let's go talk to that person. Hey, how did you get involved in this? And then all of a sudden, she finds something out that she didn't know. Like she said, she enjoys doing that, and we find either we're deviating from a policy or procedure or the policy is advancing and improving without management. Directing it that way, sometimes that happens too, and instead of saying ohh well we need to stomp that out and correct. It is part of our corrective action process. Maybe the corrective action process. Is wow. Well, our staff figured out a better way to do it. We're gonna implement that and.

00:03:05 BRIAN: We're gonna codify it and put that into the policy or procedure. So that's another way you can benefit from the internal audit process. And I I think that is usually what happens most of the time is that, you know, management does their best to come up with sound.  Reasonable policies, but sometimes staff figures out a better. Away and instead of fixing it, you go with it. So that is my recommendation for other managers out there, Kim.

00:03:31 KIM: I will say that's generally how I'm involved in the process of the internal audits is that someone's like, you know, a quick thing of like, oh, do you have a couple minutes to talk about this, Tracy? And then like, and then I'm like, yeah, this is how I'm doing it. This is how I'm involved. Generally a last minute thing I seem like for my personal involvement of like. Well, why are you involved in? This what do you do with it?

00:03:55 TRACY: And I'm glad you guys are bringing this up because I do not stick strictly to that initial agenda. I do prepare. Ohh, I would call it a loose agenda where I and I know who I'm gonna talk to at whatever times. But as as Ryan and Kim pointed out, we always stray from that because things come up in conversations. So I don't want people to think they have to stick to a a strict agenda.

00:04:15 TRACY: For the audit, it just an agenda helps me keep track of the flow of things.

00:04:21 KIM: Yeah, I think you've said previously or someone had said somewhere like you follow the audit trail and that is always stuck with me when you know during those processes. So you have the agenda, but you follow the trail. If you have to dig deeper into something, do that and don't let the audit agenda restrict what you're actually auditing and just. Go with the flow.

00:04:42 TRACY: Absolutely. And Brian, we were talking about personnel involved in the audit and this is something that we had not done in in previous years, but we actually included a special. Guest during our internal audit this year, the Chair of the Administrative Task group was actually interviewed during the accreditation program Internal Audit. Do you want to talk about? That for a moment.

00:05:08 KIM: Yeah, I'm actually really curious about, I wasn't aware like, so I read these, but I don't. Remember of them all? If we're going to.  Be honest about that, I review them. But I didn't know until you just said that this was the first time the ATG chair has been involved. So what was the catalyst for including him and the internal audit of this program?

00:05:27 BRIAN: Well, the ATG chair has been extremely involved. This particular ATG chair has been the most involved of any ATG chair we've ever had. He attends meetings almost every week with us and. He is just very active and interested and I feel like we would be really missing out on the the full audit by not incorporating his involvement in policies and procedures. Now the AG Chair has always been involved in the accreditation decision making process, but we just never really audited that part of it before but.  think because of how that role has changed. Changed in the last year, it seemed more relevant to include him in the internal audit process this time.

00:06:10 TRACY: Yeah, absolutely. And he really appreciated the opportunity to be involved in the audit. As you mentioned. I know he attends accreditation program meetings quite frequently. He's very involved. And you know, I just asked him some questions about his experiences.  With working with the accreditation program, how can we improve? I spent maybe 1/2 hour with him or so and I thought that really added a lot to the audit process.

00:06:35 BRIAN: Yeah, and it's good feedback for us in general because you know ultimately we want to make sure that the members of AASHTO, the state DOT's are happy and. The ATG chair is the basically the lead representative of those DOT's. And liaison with us.

00:06:52 KIM: For those who are unfamiliar, the ATG is the ash to resource Administrative Task group and I will put a link in the show notes to an episode we did just on that so you can learn more about the ATG there.

00:07:04 BRIAN: Thanks Kim. And the next thing I want to talk about. So we've got personnel interviewed. We've also got QMS documents and records reviewed during the audit, so that it is a comprehensive. List of every policy, procedure and record Tracy reviews during the audit. So not only does she write down what she looked at, she writes down last revision date of those documents and that is helpful for making sure we're keeping up with our policy on review and updates of those policies and procedures. Which we have is one more thing we keep up with and it is challenging, but it keeps things relevant. So it's a good thing to do, Tracy. I know this is another one of those things that we didn't always do as comprehensively as we have done. But how did that one evolve?

00:07:50 TRACY: Again, just continual improvement. I'm sure if I look back at the first internal audit I did, I may have looked at 5 or 10 things total during the audit. And then you you realize over time that you're not being as thorough as you should be during an internal audit. One of the benefits of doing an internal audit is to prepare you for. Other audits that you may have from an external agency, such as AASHTO Resource, if they're coming to your laboratory to do an assessment, or in our case, our accrediting.  Ready for our ISO 9001 accreditation? We're audited by them once a year, so we don't like to have surprises during these audits. If you're doing a thorough audit and really taking your time with it, still incorporating random sampling, you hopefully won't have surprises or not many surprises during. An external audit so that focus was also helpful with me realizing that I need to dig a little bit deeper and look into a lot more things than I had been doing in the past. Plus, over the years since we developed our QMS, we've added many additional policies and procedures that we didn't. We simply didn't have documented before. So that's part of the larger list. As we go from audit to audit as.

00:09:05 KIM: Well, yeah, just so for context, you said you maybe did in the very first audit that you did internally was 10.  Documents that you reviewed in this one, I just tried to count it up really quick and it went. Over 2 pages so I couldn't. Quite count, but it was. Over 40 by the time I lost. Track. So just as a perspective, that's a sampling of the things that we go. And as you said before, Tracy, that you review the list of what you looked at last time. So you were not necessarily getting the same things before. So that's part of your preparation that you were talking about, right?

00:09:40 TRACY: That's correct. Now there are some things that I will.  Look at from one time to another time, and sometimes that's because there was a finding on that particular policy or procedure last time, and I want to make sure that our corrective action has been effective. Of for that finding from last year, so I will likely look at something like that during an internal audit and that's part of the preparation piece as well. Looking at that audit report from last year, the internal audit report and the year prior and maybe even the year prior to that to see if there are any trends in. The types of nonconformities that are coming up, but certainly from the previous Intel. Audit ensuring that we're not repeating that nonconformity again this year, we want to make sure that our corrective actions are effective and that's a part of the piece of puzzle for internal audits would be the corrective action process. Definitely that should be reviewed as part of the preparation piece for internal audits.

00:10:35 BRIAN: Yeah, a couple of notes that that I just thought about. One of the things that you document in here on your list of records and policy and procedures, sometimes it's not even an official one. So one thing on here is Peach sheets. So sometimes you know people will have their own.  Little notes or or notebook that they use to keep track of things, and it's a non standard way of doing things. But you know Tracy put it in there. It's something that she audited against and it was one of the resources that she used. So she put it in there and I think that makes a lot of sense. And again, that's another way you can find improvements there. Among your staff members.

00:11:13 KIM: I'm curious how many of the documents listed, so there's over 40. How many did you plan ahead of time and about how many were following the audit trail?

00:11:23 TRACY: Probably ahead of time. I'm taking a look at the list here, maybe 50. Main or so that I identified from our QMS to our official policies and procedures, and the rest of that. So the majority of the list are just things that.  Came up during the internal audit things that people were talking about or they were showing me website pages or spreadsheets or records of things. None of that's necessarily. Right. In our QMS there are things that kind of feed. Into that. So the majority of it definitely are just things that come up.

00:11:55 BRIAN: OK. So the other thing I wanted to mention here is you know you mentioned cars on this particular audit. So we're going over the audit report from this past year and we actually didn't have any nonconformities this time, but it's not because Tracy is easy. About us there. Was plenty to talk about other than non conformities and actually just for transparency sake, I do want to mention in the past couple internal audits we did have some nonconformities. So in 2021 we have some untimeliness of file completion, we had database errors that were not caught. In peer review that we're over, our goal not all the time, but if it happens, if we.  Don't hit our goals all the time, then that can result in non conformity and we also had a situation where there was a discrepancy between what CL was doing during an audit and. What our AAP Policy is, and that was kind of an interesting one. I'm not throwing CCL under the bus, but the issue was that.

00:13:00 BRIAN: We didn't provide adequate training for their assessors to make sure they understood the policy. So even when it is a situation where you could say ohh, well, maybe it was the other group. Or somebody on staff that did something wrong. There's always that policy or procedure or practice that wasn't carried out that should have been so. In this case it was training and communication, which I I would say communication is almost always, if not the root, cause, a strong causal factor or almost every nonconformity that. We run into. And it's not because we're poor communicators. It's that communication is very hard. It's hard. Not everybody's great at it. And even when you're great at it, you make mistakes. So we all have to recognize that to get better. And then in the last one that we had where we had an issue, it was again. Related to training. But this time it was training records. We identified that there were we were using kind of a. A not ideal records keeping system for our training records, so that kind of kick started a project to find a new and better one which is going to benefit the entire organization. And really, everybody that we deal with because we.  To be able to control our training process better and hopefully result in better training for everybody.

00:14:19 TRACY: Yeah, that's right. And those corrective actions ideally will lead to continual improvement along with the internal audit process itself and identifying improvement opportunities. And Brian, you touched on something I just wanted to mention, when you said there were no official nonconformities this year during the app internal audit.  There is no quota for findings on an internal audit. You should never go into an internal audit thinking well. I've gotta find something you want to go into that assuming everything is in conformance and you're just confirming conformance, you're not looking for something that's wrong. So I want to make sure that that people understand that now. Typically here during our internal audits, there are findings on the reports we've had a few here and there that. Didn't, but typically we're going to have 123 findings that we have to do corrective actions for.

00:15:11 KIM: So while there was. No findings in this specific one I do. See that there were eight. Opportunities for improvements listed, and I'm sure we'll go into that and two additional comments listed in that.  What is your process like or how do you determine? What's just an opportunity for improvement versus what's officially like a finding or non conformity?

00:15:37 TRACY: A non conformity is basically you're not doing something in conformance with your documented policies and procedures. That being said, if you're looking at 5 or 6 records and there's one of them that the eye wasn't dotted but everything else is in conformance on all five of those. Records you have to think about. Is it gonna be a benefit to the organization to 1?  Note that as an official non conformity and two go through the corrective action process for that one little tiny thing that was missed, the corrective action process involves a lot of time root cause analysis. Figuring out a plan to correct it so I keep that in mind. You'll see on these opportunities for improvement. There might be these little one off things or in the additional comments section. I don't like to ignore things completely.

00:16:30 TRACY: I want there to be a record that this was noted. Those one off things. However, if they're repeated next time, that could lead to a non conformity because we're we're not taking it seriously. It was noted last time we really didn't do anything about it. That's typically how I approached that. We're I'm pretty flexible.  On that, as far as being stringent on whether or not it is a non conformity, opportunities for improvement can come up and they typically come up during conversations with other people. In fact, most of those I'm looking at that list. Most of those came from other people that I talked to during.  The audit. Sometimes there are things that I feel can be improved, but you know, Brian might have noted something or I might have been talking to a QA about something and and they would say, well, you know, I'd really like to see us do it this way. Oh, that's a really good idea. Like, let's note that in. The report. Somewhere and then in in the additional comments there.

00:17:31 TRACY: We do have quality analysts that is doing a lot of behind the scenes. It work for us. He's created a lot of fantastic workflows and workarounds and and things like that. He's brilliant, but he's the only one that knows how to do those things and that's a risk. It's not necessarily a nonconformity, but the idea is that we do want to.  Get that stuff documented. So if something were to happen and he can no longer serve in his role, we have some work instructions that other people can pick up and follow now that. If something were to happen right now and he wasn't able to do those things, it wouldn't impact the quality of the services we provide to customers, but it would impact us here from a time management standpoint. So that's why I put that in there. That's risk identification that was noted during the internal audit.

00:18:22 BRIAN: Yeah, just to give people an idea of some of the things that Tracy would find for opportunities of improvement, there were a couple policies that. We had in our quality management system that maybe were assigned to the wrong person, so maybe something was a broad policy, but because I wrote it, it was housed under the AASHTO Accreditation program, when it should have applied to every program that we have. There was another one. Where we had an intern. Internal policy that really should be public facing. So what we need to do is take that internal quality management system policy, convert it into a external policy and guidance document and post it on the website so that everybody can see it. And that's just an improvement on transparency. It is not a non conforming. Situations so that those are two specific areas that Tracy found during the audit that are going to lead to good improvements.

00:19:19 TRACY: For us. Yeah. And another one I can think of. And this has happened with other internal audits this year. We had some change in position titles here.  We had people, for example, Brian was the manager of the accreditation program and now he's the director of the accreditation program. So everything that we have in our QMS. That says the. AAP Manager will do that. That needs to be changed to the AAP director. Is that a non conformity? I don't think so. Again, going back to what I said, do we need to do a corrective action on that and root cause analysis now?  But that would not be of value to us as an org.  Relation, but I don't want to ignore it, so I put it in the report and I put it in other audit reports and we're we're mindful of these things. We're aware of it, but we wanna make sure next year we don't wanna see that popping up next year. We have a full year to go through the QMS and.  Make sure those things are updated.

00:20:10 KIM: I'm also seeing these and looking at like I'm staring at these for the first time, really like I've read it and skimmed it, but I'm actually like really looking at it and. I'm noticing that some of the language you're using seems pretty formal where you're that's not necessarily how you would have a conversation or talk. About that and.  That got me to thinking how formal do the does this record have to be or how formal do you have to be when you're creating this record for your own organization?

00:20:39 TRACY: I think that's up to the individual laboratory I have seen hand.  Internal audit reports it's perfectly fine if that works for the organization, that's fine. Our reports tend to be pretty formal and and again, I think that just comes from my background of being a laboratory assessor. When I first started, I like the look of a formal report. We have to show these reports to our accrediting body ISO 9001. So I like things to be typed up nicely. I like the sentences to be full sentences. I'm very into writing and things like that, so it's personal preference. But really you can do this however you want to. I think having it typed up in a formal way, I actually have a template that I use for internal audit reports. I think that's helpful for the next person. That is the quality manager at Ashburn resource. They can pick up the reins and they already have a tool to help them create the final report instead of starting from scratch.  So this is actually part of our QMS having an internal audit report.

00:21:40 BRIAN: Earlier you mentioned the value of having somebody who is not necessarily the involved or managing the process to be the internal auditor of the process, and it almost sounded like you're getting as you get farther away from it, the more value you have. But the question I think that some of the laboratories in our program.  Might have is that? Well, if it's better to have. Somebody who's removed. On the process, why not just hire a consultant to come in here and perform the internal audit? And is it still an internal audit? When? What's that line where you have got an internal auditor versus having an external audit performed?

00:22:21 TRACY: My answer to that is nobody knows your business better than you do. If you hire somebody to come in and do an internal audit, that's your choice. Choice. But they're not as familiar with your business as your own employees are. I think this would especially be a parent when identifying improvement opportunities. If I didn't work here, I'm looking at this list of improvement opportunities in the in the comments, I wouldn't have come up with. I don't think any of those. Because I wouldn't have the knowledge of our processes here to even identify those things. So I I think that piece may be lost if you hire a consultant to do internal audit.  Certainly a consultant would be able to look at the Astro or 18 standard or ISO 9001 and and look for those requirements and make sure that the people are fulfilling those requirements. But the conversations that we have here between staff members I find to be very, very valuable in identifying improvement opportunities.

00:23:21 BRIAN: That's a great answer because I think it really shows the value of the internal audit and we do have laboratories in our program that are AASHTO credited and they do hire consultants. To help them with the audits and I just think they're kind of, even though it's it's OK because what they were doing before when they didn't know how to perform an internal audit was not effective. This is slightly more effective, but I think it's their journey like I feel like they are on their way to being able to do this on their own the right way. And hopefully that consultant can help.  Down that path, but I don't think it's a good solution for long term.

00:24:00 KIM: How would you recommend people use a third party consultant to perform internal audits?

00:24:08 BRIAN: I'd say have them guide you on how to perform the internal audit on your own. Have them help you with the prep. Show you how to do it. Show you how to ask the right questions. Not everybody knows how to ask the right questions during an audit. Have them show you how to prepare your own worksheets, how to prepare a. For what's a good way to manage your corrective actions following the internal audit? But I just I would not look at that consultant as a a real long term solution. So I want to make sure we mentioned that because the people do struggle with it and like I said, the whole reason we're talking about this is because people on our program generally do struggle with internal audit.  Sometimes they don't find the value in them. If somebody's not finding the value that tells me that they're not really doing it the best way, because it's really apparent if you're doing a a really good internal audit, you will find value in it.

00:25:01 TRACY: Absolutely, I agree. And I get that it's hard to carve out time to do internal audits if if you do them thoroughly and effectively, it's going to take. Some time it shouldn't be an activity that you've completed in 1/2 hour or 45 minutes. It's going to take longer than that. That the prep alone is gonna take longer than that. So I understand that the resources are limited and in many of our. Customers laboratories, but it's an important activity. It will lead to continual improvement if it's done.

00:25:30 BRIAN: The right way, that's right. And you're not gonna end up with the external audit with a ton of findings. If you did a good job on the internal audit, so every time I see a laboratory that has a lot of findings on their actual resource or CCRL assessment report and then I see the internal audit has no non conformities. I see a potential room for improvement there.

00:25:52 KIM: Absolutely. So what would you think question for both of you is more valuable the actual internal audit process itself or the corrective actions that result from it is what is more valuable to the continual improvement process and to the organization?

00:26:12 TRACY: In my opinion, I think they're equally important the answer? No audit process gives you confidence that you were doing things the way that you say that you're doing them in your organization and also you're identifying improvement opportunities that will help you in the future with the corrective action process, you're correcting things that have already happened, which is good. But you also are hopefully implementing preventive action measures to prevent those things from happening again. So in my opinion, they're equally important.

00:26:46 BRIAN: That's a great question as usual. And Tracy, I would agree with you. They are equally important. I. At first I was thinking that the audit process itself is probably more important because even without taking effective corrective actions, you're still gonna make some improvements. But I have seen laboratories where they have a really diligent internal auditor.  No management support. And so that internal auditor may get a lot of value, but it doesn't extend beyond that person. And that person ends up getting frustrated too, because management doesn't support the carrying out of those corrective actions, which actually leads to not improvements at all and actually decreased morale for one of your better employees, which is going to be this diligent internal auditor. A lot of times, because it's that person's not a full-time internal auditor. Usually they're. They typically have other other value in the organization. So you definitely need management support to carry out those corrective actions. And if you don't have that could lead to a decrease. In quality overall, so you have to be very careful about this entire process from beginning to end.

00:27:54 TRACY: And management support is so critical to the success of an organization, obviously, but also to the internal audit program. We're very fortunate here at AASHTO Resource that, at least in my experience here, in 304033 years I've been here, we've always had management support for our quality related.  Activities, but I know that's not the case with some of our customers. We have a number of resources available on our website. The podcasts and webinars and newsletter articles. If you're struggling with getting management support for quality related activities and helping management understand the importance and value of those activities, you may want to direct them towards some of those resources. We have available to.

00:28:40 BRIAN: You one part in comment because I know we've gone on a long time on this, so we're not going to get into the corrective action process because that's a whole different discussion. But I do want to point out.  That, you know, quality to to make effective quality improvements, you have to have a good short game and a good long game. You really have to make those little adjustments here and there, but you should be thinking long term improvement all the time. So if you come up with a big problem, don't be afraid of it. Put it on your list of things to do.  Ship away.  Got it. But you've got to think about those. Where do you want to be in, you know, next year, three years, five years, 10 years down the line from a quality perspective, don't ignore those things as you work on the the small things and the the things that are easier to carry out, but start working on them. So it's all important.  It really is, and I hope that people get some value out of this discussion and start thinking about how you can improve your internal audit process and your quality management system. Tracy, thank you so much.  For your time today.

00:29:43 TRACY: Thank you very much for including me on the podcast today.

00:29:45 BRIAN: Great conversations all.

[Theme music fades in.]   

00:29:47 ANNOUNCER: Thanks for listening to AASHTO re: source Q & A. If you'd like to be a guest or just submit a question, send us an email at podcast@aashtoresource.org or call Brian at 240-436-4820. For other news and related content, check out AASHTO re:source's social media accounts or go to aashtoresource.org.