Welcome And Today’s AI Focus

Kim Swanson 0:02

Welcome to AASHTO re:source QA. We're taking time to discuss construction materials, testing, and inspection with people in the know. From exploring testing problems and solutions to laboratory best practices and quality management. We're covering topics important to you.

Brian Johnson 0:18

Welcome to AASHTO re:source QA. I'm Brian Johnson.

Kim Swanson 0:22

I'm Kim Swanson, and we are here to talk about AI, aren't we, Brian?

Brian Johnson 0:28

Yeah, everybody's favorite topic now. Um we we are going to talk about AI, and I and in particular, I want to talk about the uh do's and don'ts, and mostly don'ts, uh, related to the use of AI in uh development of a quality management system, uh, and all of the other ancillary activities related to maintaining your

The Red Flags Behind This Talk

Brian Johnson 0:54

accreditation.

Kim Swanson 0:54

So what was the what's the catalyst for this discussion today?

Brian Johnson 0:58

Well, uh the catalyst was some red flags that we've been seeing uh where laboratories and these are particularly new laboratories to the program or new. I I hesitate to even say laboratories because I really don't know what's going on with these entities that are submitting requests and they are submitting documentation. Uh and it appears to be AI generated uh policies, procedures, records. I I'm trying to sound the alarm here for uh for laboratories that uh may have hired people who are not qualified and have presented themselves as such to carry on the work that they have been doing, uh, and for owner project owners and specifiers uh to be careful.

Kim Swanson 1:50

So isn't one of the benefits of the ASHTO accreditation model that it isn't just a desktop audit, right? We do have on-site assessments and we do have proficiency testing data that goes into ASHTO accreditation. So won't those act as the checks and balances to the problem that you're seeing?

Brian Johnson 2:16

Yeah, that that that is a great point. And yes, that our onset assessment is going to help with that a lot because we will determine if that is a real laboratory, if they are uh competent uh to perform these tests, if they uh are able to maintain a quality management system. What I'm concerned most about is that we're not the only crediting body out there. And I know that some of them really uh tout how easy it is to get accredited or how you don't have to waste time or money with an on-site assessment. Uh so really I don't know what's gonna happen with those type of situations, but um be concerned if you're uh uh um out there and hiring somebody that has never had an on-site assessment.

Why On-Site Assessments Still Matter

Kim Swanson 3:06

What can AI be used for?

Brian Johnson 3:09

You know, there are things I really um want to stress here. Um, first of all, we accredit to ASHO and ASDM standards.

Derivative Work Bans And Copyright Risk

Brian Johnson 3:18

Those are the standards that people are generally using that are accredited through our program and through a lot of other programs. Um both of those organizations prohibit the use of AI um to create derivative work, you know, work products. Like so that could be like uh an internal exam or a policy or a procedure or a template or record. You're not allowed to do that strictly using AI. Uh well, I shouldn't even say strictly, like using AI. They have a very clear policy, like you are not allowed to make derivative works using their standards. Uh so like if you are doing that, you're violating your user agreement with them, you're violating their uh their rights uh to maintain their own copyright. Uh so you shouldn't do it just for that alone.

Kim Swanson 4:14

We aren't even able to do that um as employees of AASHTO n. So this is not like when our staff is making worksheets or policies based on the standards, it's all human-made.

Brian Johnson 4:32

Yeah, right, right. So so I mean, there are some there are some like legal and ethical concerns right up front, but yeah, you can't upload anything. Uh you can't ask your AI agent or or platform to create anything uh related to the standards. Um I'd say like this goes even as far as like don't like do not ask your AI agent what a reasonable test result would be, because I suspect people are doing that too. Like, I think that people are just completely like there is no like some people have no ethical obligation in their mind on how they use this because they think, oh, isn't this so great? You know, what a great tool. It can make my life so much easier. It's like, yeah, there's some things that can make your life a little easier on. Like, for example, I use it in Excel all the time now because I am terrible at writing those formulas in Excel, but I can ask Copilot to do in layman's terms, like what I want it to do. And most of the time it does it. I still have to validate it, though. I think one of the things that people were excited about as a possibility is making checklists. Everybody's always asking us where the you know, can you give us the checklist your assessors use? Uh, I think that some of them think like, hey, it's no problem. I can just have it make a checklist based on the standard. You can't you can't do that because that is a derivative work. Um, so really you can't can't do a whole lot. So sorry. Like I know I know that's a this is a wet blanket episode for a lot of people who are very excited to just start using it.

Kim Swanson 6:10

Yeah, so all that stuff you can't do. Are there examples of things that how it can be used? Sure.

Safe Uses Like Formatting And Shuffling

Kim Swanson 6:18

That you yeah.

Brian Johnson 6:19

Okay, so let's I think one of the one of the things that would be helpful to people. Um, you know, a lot uh we we find a lot of a lot of laboratories will write exams um for their technicians, and those are those are used for multiple reasons, some of them just as a a way to to check to make sure that they know what they're doing. Um, but sometimes it's for conformance to certain standard requirements, like uh ASTM D3740 that requires the laboratory technicians pass a written exam. Now, one of the things that people struggle with is keeping those exams up to date or shuffling, kind of shuffling the deck so that they're not like, oh yeah, answer to number one is always C, answer to number two is always B, you know, whatever it is.

Speaker 2 7:03

Yeah.

Brian Johnson 7:04

But like I think it would be reasonable if you made your own exams, if you went into like Word or something and you said, hey, Cobilot, shuffle these for me. And like, of course, you're gonna have to like make sure that they did it right. Um, but like maybe that's a an easy way, and and maybe you can sure you could have shuffled them on your own, but you could you know you could save quite a few uh minutes using using AI to do that for you. Um, but you can't say like, hey, here's a standard write an exam for me.

Kim Swanson 7:35

Yeah.

Brian Johnson 7:35

Like there's a big difference between those two things.

Kim Swanson 7:38

Yes, yeah. Helping you for uh format something is definitely different than writing something.

How AI Documents Give Themselves Away

Brian Johnson 7:45

Yeah.

Kim Swanson 7:45

How are you so sure we're getting sent or the accreditation program is getting sent uh AI documents and policies and stuff? Like, how do you know this?

Brian Johnson 7:58

Well, um I can't know, but there's some real uh obvious indicators. Uh for example, uh, I think one of the first ones we saw, it kind of made us all laugh until we started seeing more of them. But it was uh it was a procedure for checking uh a piece of equipment that had a uh as required equipment, a five-gallon bucket. And there is no five-gallon bucket that is relevant to any of what was going on, and so we were kind of laughing back and forth about it and said, you know, what would they even use it for? Um then we got another one of somebody uh standardizing a thermometer, and one of the required pieces of equipment was a shovel. Why would a shovel be used for that? Who knows?

Kim Swanson 8:51

Not only did they use uh AI, uh they did not look at it. They didn't need to be able to do it.

Brian Johnson 8:58

They did not look at it, they just they just went with it. So a couple other examples are uh now here's one where somebody did probably look at it, maybe. Okay, or maybe it gave it very specific instructions. Okay, because what they ended up with is a multi, I think there were five or seven tabs associated with this Excel record that was for something very simple. Their AI agent made it extremely complicated and time consuming and also nonsensical with all of the details and all of the things that these people had to do. So they took something that should have been a one-page simple thing if they knew what they were doing and turned it into a nearly impossible task using AI.

Kim Swanson 9:47

Okay.

Brian Johnson 9:48

So that's one thing. Other ones are uh, you know, I mentioned like most of these records are typically like a you know one page long, maybe don't even fill up the whole page. And we'll get an AI generated procedure and record that's you know seven pages long.

Kim Swanson 10:03

Oh wow.

Brian Johnson 10:04

Um all sorts of irrelevant information. So like there's some there's some obvious indicators that we see. Um, but what in those cases, what I've uh when when my staff asks me, you know, what do you want me to do with this, I say just reject it. Uh because you get you can tell they they're they've put no effort or thought or care into creating it. I don't want my staff spending their time giving them very specific details about what needs to be fixed, because what what's gonna happen is that the format of that is not gonna change. They're just gonna change a couple details and they're gonna be left with something unusable and nonsensical still in the end. So I would rather them just start over in those

Public Safety And The Cost Of Bad Data

Brian Johnson 10:52

cases.

Kim Swanson 10:52

Why do we even care about this? Why is this such a problem?

Brian Johnson 10:59

The contractor is using AI to generate, you know, whatever they're whatever they're making, whatever they're building. Uh let's say that they started using AI, and then the testing lab that's supposed to be checking them is also using AI, and nobody knows what they're doing. And then you have a problem, right? For everybody, uh whether they're involved in it or not. You know, anybody who's using whatever that end product is could be at risk. So uh it's really not a good idea. And um as I mentioned before, I'm concerned about the credibility uh of these agencies, these testing laboratories that are coming into our program. You know, when AASHTO accredits a laboratory, there's an expectation that it will be a knowledgeable, competent testing agency that uh the DOTs and other um state, federal, local agencies, private uh private entities that require accreditation can rely on. And if they don't have the care or the knowledge or the understanding or the professionalism that we would expect from a national accredited lab, uh then that really hurts our reputation as an accrediting body, and and we don't want that. I like I said before, I mean, we're we're seeing this more with new companies coming in that that don't have any sort of reputation. But uh it's probably happening at other companies that are established. Um one of the things we I care about is we uh we always struggle with timeliness. Uh, you know, we have a a lot of accredited labs in our program, and our and our staff does their best to try to keep on top of things. Um, but if we're having their time wasted on reviews of of nonsensical documents, uh that's not the best use of their time. So I really don't want to see them uh spending time with with those customers um until those customers are are willing to get serious about what their obligations are.

Kim Swanson 13:00

You kind of touched on it briefly, but it is really a public safety issue for buildings, for bridges, for roads. Um, if the data and the materials that go into it are not accurate or at, you know, not measured correctly. Um, that's really a big broader concern outside of you know the Astro Accreditation Program. That is a public safety issue.

Brian Johnson 13:26

It is, it is. And I and I and if you're listening to this and you're like, this is absurd that you're even worried about this, like it doesn't matter that much. Um, I I want to just remind everybody that before AI, we had these problems already. I mean, human intelligence has also got a lot of faults with it. Yeah, and and some of those faults are what are leading to this problem. So imagine, I want you to imagine the same kind of problems that we've had with humans, except replace those humans with people who know nothing. They don't even know that what they're doing is wrong. They they have like no concept of what's going on because they have faked it till unfortunately they made it to a position that they should not be in. And and that that is not it, it's just gonna exacerbate the problem. Uh, it's going to uh you're gonna have more of those failures, you're gonna have more of those failures that you don't know about till it's too late. Because if you have everybody in the chain not understanding why they're doing what they're doing, um it's just gonna be a bad result. And and like I said, too late by the time you know that it's been a problem.

AI As Record Falsification Amplifier

Kim Swanson 14:41

This use of AI is just another branch of the falsification of records, right? We have always seen some organizations that lack integrity or individuals even say that lack the integrity and just kind of write stuff down or kind of just throw things together without really thinking about it, without knowing what they're doing, and just kind of copying records from a friend or colleague. You know, like that's AI is just making it easier for people who don't know what they're doing to submit things that they think shows that they know what they're doing when they don't. So like this is just this is just uh an amplification of an existing problem that we have been seeing, but it is more readily available. Um and I think because it's often seen like you know, entering standards into AI and uploading things like that seems like a victimless crime, so to speak. It makes it a blurrier line for some people. Um and I think that's why we're seeing it a little bit more.

Brian Johnson 15:54

Yeah, and I I really wish people would have more of an understanding of like the theft of intellectual property and and like I guess if you've never made anything, then it seems victimless, right? Like standards development. I mean, we've talked about how painful that process can be and how much time and effort goes into the creation of those standards. They should not be free and they should not be stolen. If they would be, then there no one would put the effort into making them. And I know a skeptic might say, Well, who cares? Like maybe we don't even need those, but it's like, okay, well, imagine a world where there's no clear instruction on how to do anything, and people just do whatever they want, right? And how would that go?

Kim Swanson 16:39

I mean, we do know how that went. That was before standards, everything was unregulated. We do know how that went. Just go back in time. We know exactly how that went. Now just put it with the filter of the fast-paced modern technology. I mean, we also have unregulated artificial intelligence, you know, in AI right now and data centers. And we're going through that whole, you know, need for some standardization in that in that area as well. So we've had past episodes on AI and automation and like I think there's no way to get around the future that will include using some artificial intelligence and machine learning for sure. I think we are not suggesting that that's not gonna happen, but I think we do need to be really smart and encourage our listeners to be really smart about how you that you have smart people using AI. You know what I mean? Like if you are using it in the ways you're allowed to use it in this instance, um you still need the human check. Like that you still need someone who knows what they're doing to review it and can't just pass it along.

Brian Johnson 17:57

Yeah, you're right. It's an incredible valuable tool, and we have to figure out how to use it without violating uh other people's

Checks And Balances In Accreditation

Brian Johnson 18:05

rights.

Kim Swanson 18:05

I do want to kind of circle back on how the Astro Accreditation Program is using uh the inputs from the on-site assessments and the proficiency sample program testing to uh prevent these types of entities that don't really know what they're doing from getting accredited. Like how how are can you explain if they were doing anything to make sure we are catching these uh people and entities earlier in the process, or do they just have to go through the process and it's at the accreditation step that we notice it?

Brian Johnson 18:46

We have an application process and one thing we want to do before we go on site is get a copy of their quality management system just to make sure that they have one and they understand what that is. Because we don't really want to waste time going to visit a laboratory that isn't in a laboratory or doesn't have um the capabilities of doing something that we would need to see for them to get accredited. Um so we have a little bit in that step, but I think we can do more. Um, but where it really happens is what you talked about earlier, the actual onsite assessment. So we go there, we have an assessor show up, they audit the quality management system, policies, procedures, records, they check the equipment, they watch the technicians run the tests. Every single test that's listed on the AASHTO n accreditation directory, someone at that laboratory performed it, either to our, you know, to the satisfaction of the requirements of the standard or made some mistakes and submitted corrective actions. Um and the last part of it is the proficiency samples. Uh, so we do send those proficiency samples out. Um this is one of the areas that I worry about uh because we we're not there to watch them run them. Um and we have caught people faking their test results before, and it continues to happen. And we do uh we we do our best effort to catch those situations and prevent them from happening. So in some cases, those laboratories kicked out get kicked out of the program. And in an extreme case, that has happened. Um most of the time it's a it's a case of a person doing the wrong thing and that getting corrected, and then they have to get a blind sample or something like that to show that they can really run it. But there are tons of, I'm not gonna tell you all the indicators we have for that because I don't want to give it. I don't want to give somebody the how-to on how to falsify uh proficiency sample records, but uh it it can be easy to spot. It usually is relatively easy to spot because they do such a poor job hiding it.

Kim Swanson 20:53

We've had, as we mentioned before, we've had similar issues with people falsifying records and data all the time. We this has been since accreditation began. Someone has always tried to work around the system. Um, but now it's just becoming easier.

Brian Johnson 21:09

Yeah, it's just easier and they have a better tool for it. So again, it all comes back to the failure of human intelligence. Like, this is all like AI is just a tool as far as we're concerned. Like, that is a tool they can that that can be used for certain applications, but what we've been talking about is not where and how it should be used. So, like, we want to make sure people understand that.

Kim Swanson 21:32

But again, we do have systems in place and checks and balances within our program to make sure that we can keep the integrity of the AASHTO accreditation program uh where it's at.

Brian Johnson 21:46

Yeah. Yeah. And if you're out there wondering if we're seeing an increase, yes, we are seeing an increase. I actually we've uh, you know, as as the technology has rapidly increased, so has the amount of instances of these situations rapidly increased. So we hope that somebody listens to this and is like, wow, that's interesting. I'm gonna keep an eye on what's going on at my laboratory or with these laboratories that we're hiring to make sure that they are not uh falling into these traps.

Kim Swanson 22:18

Mm-hmm. Yeah, uh, as the saying goes, one bad apple can spoil the bunch. Um, so if you aren't hiring correctly and hiring people with the critical thinking skills and the experience that they say they have, um I think you are more at risk for for something like this to happen. And then again, the chain reaction of what this impacts of the falsification of records um and policies and procedures is not good.

Brian Johnson 22:47

No, not at all. If you have follow-up questions, let us know. Uh, we're not gonna be able to share the exact records with you, but you know, I'm happy to talk about. And actually, I think we're gonna cover this in the upcoming uh technical exchange that's gonna be in March in Kansas

TechX 2027 And How To Reach Us

Brian Johnson 23:02

City.

Kim Swanson 23:02

So And that's a great segue into the us celebrating 10 years of TechX. Um, we are in the planning stages of the 2027 Astra re:source Technical Exchange. Um, it's in the heart of the country, uh, Kansas City, Missouri, um, March 8th through the 11th in 2027. Um we are in the planning stages. We are hoping registration will open um sometime in the winter this year. So hopefully by December, but maybe January, um TBD on that. But you can find more information about it at ashtoresource.org slash events. And I'm looking forward to um all of the things that you're planning. But we will have, we will likely have some AI uh panel of some sort. Um, I believe. But again, you can check out ashtoresource.org/slash events when we have a draft agenda and all that in the next few months. We'll post that there. But thank you, Brian, for taking time to talk about this day. Hopefully it was useful for our listeners.

Brian Johnson 24:08

All right. Thanks, Kim.

Kim Swanson 24:10

Thanks for listening to AASHTO re:source QA. If you'd like to be a guest or just submit a question, send us an email at podcast at ashto resource.org. Or call Brian at 240-436-4820. For other news and related content, check out AASHTO e re:sources social media accounts, or go to AASHTO re:source.org.