Okay. Good afternoon or good morning or good evening, depending where in the world you are. And thank you for joining our webinar, A Watchful Eye, Regulators Double Down on AI and Compliance. And, I was thinking there’s kind of a trick in the title there. Right? Doubling down on AI and compliance. Or do we mean the compliance of AI? Right? Or do we mean how AI can be used in compliance? Right? And so we’re gonna talk about both today. We have a really great panel, and we have a lot to cover. And I’m just gonna pick from just whoever I see first on my Zoom. Just do some introductions, and we’ll dive in. So, Alvin, I see you, then she’ll meet. So, Alvin, who are you? Hey, Alex. Very nice to, to be here. Thanks for having me. So my name is Alvin Alvin Juan. I’m on the market development team for financial services here at AWS, with a focus across, machine learning as well as generative AI. Prior to joining a so I’ve been with AWS a little over six years. Prior to joining AWS, I spent twenty years within the financial services space, most recently at JPMorgan where I was on their compliance surveillance team. So this topic here is near and dear to my heart. So, again, very happy to be here. Thanks for having me. Amazing. Shlomit? So hello, everybody. Nice to meet you. I’m Shlomit Labin. I’m the VP of data science at Shield, a financial compliance. Have been in the AI industry for many years now, leading, AI driven solutions in many domains and in the past few years in the financial, compliance domain. So, we’ll talk about that today. Cool. Amazing. We’re waiting on Laura Aguilar, who is a data scientist, heads up, I think, a data science, function within communication surveillance for Citi. Myself, I’m Alex Delucena. I am Shield’s, director of product strategy. Prior to that, I worked in compliance myself. I, headed up a com comm surveillance function, and, now I’m on the vendor side. So, I wanna start off with sort of a thought to help kind of frame our conversation. Right? And it’s kind of a contradiction that sits at the heart of interest and adoption of AI in the compliance space. So on the one hand, it’s safe to say interest is elevated. Right? And I think a lot of that extends from our consumer experience. AI is already such a part of our lives, whether it’s suggesting what movies we watch, what music to listen to, clothes to buy. Teenagers use GPT to GPTs to do their homework and so on. Here’s Lord. So, no no no worries on that. Lord, why don’t you quickly tell us, who you are? And then I was just kind of framing our convo convo, and and we’re gonna kick things off. So, who are you, Lord? Well, that’s the eternal question. Do we really know? Anyway, Laura Aguilar? You’re still I am, director of, surveillance design and research within, the surveillance intelligence unit of Citi’s ICRM surveillance. So I had a group of data scientists who look for ways to innovate, improve our, surveillance design for markets, info barriers, and electronic communication surveillance. Amazing. And I I think, so that rounds out our panel. And so just jumping into where I was. So I was talking just to help frame our conversation today about a contradiction that I think sits at the heart of of of, adoption of of of AI in the compliance space. So we can all agree interest is elevated. Our consumer experience is already very embedded in the use of AI, and I think that kinda carries through. Right? People are asking, how can we use these capabilities in a meaningful way to improve compliance? Right? At the same time, interest in compliance of AI is equally elevated. Right? And they don’t contradict each other, but there is some tension between the two because a lot of firms aren’t sure, how to vet these capabilities in order to adopt them. Some have MRM functions. We see regulators beginning to have, to lead, whether it’s the EU AI act, recent FINRAGen AI guidance, and that kind of thing. Right? So we have these sort of two elements. And then there’s something in the middle that I think gets talked about less. Right? Which is, how can AI actually help? Right? So there’s interest. There’s compliance. But how can it actually help? What are good use cases? What’s smoke because there’s a lot of smoke? What’s real? So in this webinar, we’re gonna look to cover all three. Right? What can AI do? How do we vet it in such a way they can be compliant? And what are the best use cases and best practices for adoption of AI? Let’s start with present day. Let’s baseline. Let’s talk about where we’re at. And so, Lore, we’re gonna start with you. Within your function, how are you using AI today? Sure. Thanks, Alex. So I’m gonna put in the context of communication surveillance, which is really We know we could we could step back, and it’s been used for ten and all that kind of stuff. But within your communication surveillance, how are you specifically using AI? Sure. I’m gonna, I’m gonna start with, like, the broader definition of AI, not necessarily Gen AI. I mean, we have been using the traditional AI in terms of machine learning for, for our, alert generation. So it’s been useful in supplementing, complementing our as folks here, probably know, the workhorse of, communication surveillance is lexicons and keywords and rules. So, and as is probably as well known, the, one limitation of that approach is kinda missing the context of a communication. And that’s where potentially NLP and, some of these more traditional supervised learning approaches can help in providing more context into what you’re looking for. So that we’re actively using, for for your generation. Yeah. Cut in really quickly. And and for that that traditional flavor, that classic AI, I think that typically means you’re taking city comms, city sentences, and then you’re looking to, build a model, you know, that focuses on a specific behavior from out of those sentences. Right? And that’s very iterative process where you’re kind of saying, this satisfies it, this doesn’t, and then kind of looking to do it. Right? Right. Right. Absolutely. And Right. And and you can also use that in assisting for case disposition. If you have a historical corpus of what has been fruitful alerts, you can train on that to look for more of that. Now so that that is already in place, and there’s already a model risk, framework around that. Now with the, with this newer flavors of generative AI and the explosion of LLMs, I mean, the promise of that where we’re looking at that is, again, in the alert generation, but the ability to, significantly account for, for context and attention that even your, you know, traditional, say, LSTM approaches, some of these traditional supervised learning, approaches just wouldn’t be able to do. Right? So that is that is big. The, in the area of classification Mhmm. Summarization. I think we might go into that a little bit. Yeah. Yeah. So we’re we’re gonna touch we’re gonna touch on use of LMS, classification, summarization. So so that’s all. But but finish your thought. But high level, that’s that’s, an, that’s what we’re currently exploring, classification, summarization, and then we’re also looking to explore information extraction, topic modeling. I mean, you can imagine if you’re in the comms surveillance space, those are, you know, key use cases. Okay. Excellent. And I see I should say that if people wanna ask questions, we have a lot to cover, but, obviously, you know, we we’ll try to answer some questions too at the at the, end of the discussion. So, Alvin, over to you. Right? I think what a lot of people know or know to different degrees is that, AWS and other big tech companies are more and more sort of embedded within banks. And so what’s happening is a lot of banks are sort of building these enterprise agreements within AWS, and then AWS has a suite of tools that they can use, that banks can use to sort of experiment with and and and and build their own solutions or or or that kind of thing. So just from your perspective, what kind of tools are people using, and what are some real, you know, some real world examples of how kind of we’re you know, keep with that present day lens. Right? Kinda Laura walked us through that you see firms using, AWS kinda more broadly, but I think still as best we can within the kind of compliance, space. Absolutely. And and maybe I’ll take Laura’s example and and just expand on that a little bit. Right? If you if you kinda take a look at that end to end surveillance process. Right? Now this can be both trade surveillance as well as ecommunication surveillance. The overall process is is relatively the same. Right? There’s effectively kinda two sub the the alert investigation side. Right? You research the alert, and then you close it either as no action required or you escalate and potentially file SAR. Now to Laura’s point, the use of AI and machine learning traditionally has been very focused on that first area, right, around the alert generation piece. Right? So for example, we see machine learning being used in anomaly detection for something like insider trading or pattern recognition for something like spoofing and layering. Right? Now the key problem to solve here, right, it’s really an optimization problem. Right? You’re trying to reduce the number of false positives, that are generated while not missing any actual fraudulent activity. And machine learning is very, very good at solving that type of problem. Now where it gets really interesting over the last year and a half, right, is just the advancements we’ve seen in generative AI. So now we’re seeing financial institutions leverage GenAI to make that second part of that process, the alert investigation process more efficient. Right? So I’ll give you a a specific example here. One of the global exchanges has integrated GenAI into their market surveillance technology platform so that instead of an analyst having to go through and manually search for and collect all of that necessary data that you need to, disposition and alert, right, what this Gen AI application will actually do is automate and produce for you in a single pane of glass things like a consolidated table of the company’s regulatory filings, right, new summaries, links to companies, sentiment analysis, and more. Right? And they found that by reducing the manual effort involved, they actually saw a reduction, of around thirty percent in the time it takes to, close those alerts. Okay. That’s super interesting. I think and let let’s just but and and I wanna start to start to explore, like, the potential more, but I I think just to round things out, Shlomit, let’s go over to you. Obviously, you and I both work for, you know, comp surveillance archive and and comp surveillance vendor. And and and sort of our businesses is to kinda try to add value to these, to the use cases around regulatory compliance and and that kind of thing, but also to think ahead, right, and to think how these tools can also be used to improve people’s lives, to improve their compliance, and that kind of thing. So why don’t you talk about, you know, what you’ve worked on today? And I think nice transition, and people can jump in, and that kinda goes for the whole conversation, sort of jump in where you feel you need to. You know, let let’s start with present day, and then maybe let’s start to lean into potential. And we can start talking about how we see you know, where we see GenAI going, AI going, and how we can kinda use that stuff. So over to you. Sure, Alex. Maybe I will, ignore your question and do a little bit of zooming out, in the terms of, looking at, development of AI technology in the past few years. A law has described very well the challenges and the common practices of the compliance or comm surveillance. We are looking for detection. We’re looking for, maybe summarization will be added. Topic analysis, these are the common challenges that everybody are dealing with today, and technology is constantly improving, maybe pattern analysis and machine learning now, maybe gen AI, and we can do everything better, again, as Alvin was mentioning here. Next phase is, again, investigation. And therefore that, what we’re trying to do is I’m just repeating everything that was mentioned here. Also in Shield, collect information of on a multiple, amounts of data points and produce meaningful insights from it, multiple communications, reporting, all kinds of analysis to provide it automatically, look for things, suggestions, for follow ups, provide all of this, for the compliance officer to pursue any kind of, risk that may have been hinted. But maybe to take a step back and to talk really about the breakthrough in AI technology and in Gen AI and what does it now offer that, it didn’t offer before. So we know that, again, the real breakthrough that started with the chat GPT and all of the models that followed it was the real jump in human level reasoning. The ability to understand much deeper and a much more complicated issue. So even if we’re looking at a simple detection task, the understanding of the full, nuances that occur in a communication can be done by a single model placing it together. The second thing is that, generative AI is now, communicational. So we can actually theoretically talk to it or interact with it, and more importantly, perform multiple steps, in our assignment with a kind of assist that follows that as around, that remembers the steps that we did before and also can integrate all of the information and all that was done so far with additional things in the neighborhood. I want to say one thing. The breakthrough in this technology has created the first real, I think, impression on everybody. There’s something very new that we can now use in any new capability that, we really need to adopt. But the second thing was and that’s your question, Alex, how can we adopt it usefully? And, you know, there are all kinds of trials in this, domain. Seminarization is a good example. This is something that these engines are doing well. Is this the full benefit that we can do from these models? It’s probably the first steps, and we can, leverage them for a much more advanced, challenges. Right. Let let me stop you real quickly just so we can kinda take a gradual approach and and and and maybe talk about summarization a little bit first because I think Laura raised it as well. So I think so one thing that’s been interesting to me about reading some of the positions or or or or that regulators have put out so far is that they’re really doing their homework. Right? So if we see in the EUAI Act and we see in the recent FINRA guidance, we see that with the FINRA guidance, for example, they’ve taken this position on the one hand where they’re sort of saying, the controls you have are good enough. Right? Keep going. Right? And they list all these examples of how AI could be used to improve compliance. And within the space of compliance surveillance, specifically, they list detection, right, which we started to talk about a little bit, and they list summarization. Now they’re not saying run into it, but they’re saying these are examples where we see it being used, and we’ll get into the controls later in the conversation. But, basically, the controls you have are good enough, you know, to sort of extrapolate the information you need to be comfortable with it. Right? So let’s assume that summarization is something that people could lean into today, and and and let’s think about how it could be used in a meaningful way. And then also, you know, because you were just starting to go down that that that lane, how it might just be the first pass and and and and maybe there are limits to it. And, obviously, Laura, jump in, Alvin, jump in, you know, that kind of thing. Maybe I’ll start, and, Delo, you can add because you’re also experimenting with this. I think that the challenge in summarization is you can provide the summary is something fluff. It’s not a yes, no answer. You can provide multiple, summaries of the same conversation. Some will be more useful. Some will be less useful. You know, the common answer that we give in the case of the compliance officer, summarization of a conversation can be lacking the specific risk that, occurred in the communication because it’s not the gist or the essence of the the, overall talk between two people or four people in the communication. So the question is, how do you direct the summarization to actually be beneficial for you and for your specific needs? So we now have this great technology that is very good in creating summaries, but it needs to be directed to answer your specific needs, which in our case, I think all three of us is, to find financial risk or to detect the financial risk in our communication, either by ongoing alerts or by investigation. So, it’s not, you can ask ChargePT here is a communication, please summarize it for me. It will give you an answer. The technology is there. But, in order to make it useful for the specific use case, you need to do more manipulation on it to fit it with more information, to make sure that you have more controls in place, so it will actually serve your needs and not just produce a nice summary. No. I I think that’s a really key point, in in terms of what we’re using. I mean, one one clear use case is, let’s say you have one, again, an alert, that, communications alert, and it could go on for, you know, multiple, multiple exchanges in a chat. That could be a, and it will be a mix of certain things. Some will be talking a little bit more business, something you’re interested in, and then just like that, they’re gonna delve into something more, casual. Right? So how is if if we’re able to maybe prompt it in the right way, so to summarize it in the right way and kinda like what we’re looking for, that’s gonna that’s gonna assist the, the, the surveillance analysts who are looking at this. Because remember again, I wanna put it into the context of what folks are used to and where we’re moving. With the lexicon, approach in in in surveillance, you know exactly white flag. Right? Because the words are there. Here, it’s more of what’s the, what’s the context. Right? What did it find? Right? Perhaps in that summarization, there’s be a little bit more, explainability or reasoning or into at least what you should initially look for, and that will help in the, investigation process. Right. I I think one thing that people need to ask themselves with summarization is is how it’s adding value. Right? Is it adding more work? Right? Because now you have two steps. Right? Or is it or is it creating less work? Right? Because ideally, that’s what it should do. And I think we’re not at a place yet. Not that we’re not at a place yet. I think it’s really just how we vet these capabilities and the specific use cases they’re funneled through, right, as we decide them. Right? Because you might have two sets of data side by side that you now have to review both together together, or you may have a piece of data that provides better at a glance insight that allows you to sort of make a quicker decision. Right? Which is the ideal. Right? I think another thing I’ve seen, and this one’s a little harder to vet, right, is how do you vet because let’s just say that because there is just so much out there, how do you vet that something actually works for you? Right? Could you just plug this into ChatGPT yourself and ask it a piece of trade jargon or something like that, and it’ll tell it back to you? Or is this something, you know, that that that that that that whatever capability you’re being sold or experimented on, you can meaningfully add value to? Right? And I think, Laura, you have a privilege there, right, which is that you’re already living deeply inside your use cases. Right? But I think, Alvin, just to kind of spread it around, like like, since your view is more generalized, how do we help, you know, how do you help firms kind of zero in on, we’re talking about summarization, but, you know, like, as you see fit, like like, how do they kind of make the decision they need? Right? You know, how do they sort of vet these use cases in a meaningful way so they can use these tools? Yeah. I mean so largely where we see customers leveraging, something like generative AI, right, are more around the internal facing use cases. Right? So, again, it’s almost like the the Ironman analogy. Right? The the Gen AI is the suit. Right? And you’re making the person better. Right? So, if you need to get information, the Gen AI can grab that information. It can summarize it. It can collate it for you. Right? So you don’t have to go to three different data sources to get that information together. Right? If you need to, say, summarize, say, a news story or maybe an earnings transcript. Right? It might be eighty pages of earnings transcript, and you wanna summarize down to three paragraphs. Right? It’s very like, to to, Shalomie’s point, it’s very, very good at doing things like that. Right? And we’ve seen, that it’s added value. Right? The example I gave before with a exchange, that that’s doing this today, right, it is reducing the time it takes to, evaluate and resolve those alerts, right, by by thirty percent. If we kinda venture outside of, surveillance, right, things like within investment banking, right, like summarizing historical research, things like that, generating new ideas, absolutely. Right? You see kind of that summarization, being done. Now kinda to your point, Alex, is how trustworthy is it. Right? Can you trust that summarization? And I think we’ll touch on this later. Right? But there’s certain techniques there now, right, that have been developed, which really links kind of the output, generated by the Gen AI right back to the original sources. Right? So you have kinda trust, within that system. Cool. Amazing. And then I think, Laurie, you had started to talk earlier about LLM classifiers, right, and how you see those, potentially change in the space and how you detect and that kind of thing. So we’ve talked about summarization being one use case. You know, how do you see sort of the use of LLMs as classifiers are even more broadly changing the work you’ll be doing in the next year or two? Yeah. I mean, in when we’re, building surveillance scenarios, we’re really putting pieces together. And you can have classifiers that, that detect certain, indications of behavior. Give you an example. Secrecy. So how do you how would you kinda detect a note of secrecy, in a text? I mean, there are, the traditional machine learning is you feed it through some examples. Right? But maybe the promise of this as a classifier is, you know, is it able to detect that better? Right? Detect the nuances in the tone in the conversation better. Right? And we’ll we’ll see once we, once we actually try the, the the classifier. Nice thing is we have a number of, examples and ground truth to, to compare this with. Plus you also have a challenger model in your existing ML already. Another one is topic modeling. So, again, in, again, in the realm of sim surveillance, the very simple question of, is this trade talk that’s going on, or is this more casual conversation? Right? Is this deal talk, or what other topics are being clustered around in this in this conversation. That, again, I mean, we have techniques that try to do that, but I think the the the exciting thing is we think that LLMs, because of the the corpus they’re trained on, is potentially gonna be better able to do this. But one thing I’d I’d note in both summarization and all of these use cases, including classification, I mean, just just be mindful of the same time that we’re looking for, LLMs to be able to tie the story in a chain of conversation. We also have to take, to look at whether the accuracy I think, Alvin, you you you alluded to that. The accuracy, how does it relate to the length of the input text? Right? Is it you know, the longer the thing the is there some degradation in the accuracy the longer you give it? Right? But then that’s just one con consideration that I think folks should be mindful of. So let me pause you right there because I think that’s a good way for us to kind of start talking about controls, governance, and how some of this stuff can be compliant. Right? So I think just a a quick thought to phrase that is is we have two parallel functions. Right? So a lot of firms have, and in in a sense, they’re both compliance functions. Right? So we have the model risk management function, which doesn’t sit in compliance necessarily, but is used to vet models across a firm. Right? And certainly, Laura, you’ll have been through that process of getting your models through those teams and for them to sort of go through the criteria they need to to feel that they can be comfortable with what that model is meant to do, right, and the outputs it gives and and that kind of thing. Right? Right. And then we also have the question of compliance, which is just, like, how do compliance teams become comfortable with just the idea of AI? And and and and within some banks, we see they now have AI policies and that kind of thing. You would think that these would be sort of parallel in a sense because they have similar goals. But what we’re seeing is is on the one hand, the MRM functions are quite mature, but haven’t quite adjusted to how they conceive of how they should vet LLMs. Right? In part, because of the supervisory model, you know, people like yourself, Laura, have given them. Right? Where they they’ve sort of come to understand, like, okay. Well, if it comes from this dataset, then I get it. Right? And I I can kinda follow through how you came up with these answers. Right? So back to you, Laura, and then, you know, we we we can all jump in. Like, what has been the hesitation internally, right, around leaning in more to the use of LLMs and that kind of thing. Or maybe it isn’t and there isn’t one and there’s a gradual approach. But what what would be some of the considerations for you internally as you look to kinda lean into this new world? Absolutely. I I I would say it’s not so much hesitation as, caution, as, reasonable caution. There is there are no risks in, you know, generative AI, hallucination, privacy, potential fairness and bias on, on the data it was trained on, especially if you’re not on the corpus it was, trained on. Data limitations. It’s it’s trained as off this particular date. Right? So the accuracy is is conditioned on that, that have to be navigated. But I would say there’s general openness to, to, adoption of this. And the MRM, piece really has to take a look not not just at your, the traditional things you’ve been looking at, soundness, you know, again, potential bias, but but these other considerations, you know, hallucination is the is is a big one, because because transparency and explainability is another one. We know how to do that for the more traditional ones. I mean, how do you do that for, for for that being less true for a classifier that you’ve sort of taken a big, you know, Gen AI engine and then kind of, like, focused it on something in particular? Like, wouldn’t wouldn’t things like hallucinations or would not be less of a concern? Potentially, potentially. I mean, hopefully. But these are things you have to test. Right? When you say, is this thing a small talk? You really need to be able to validate it, to look at the accuracy of the output, to be smart about how you assess that, that testing, segmenting your data across different categories, make sure that, you know, there’s kind of consistency in the way it’s it’s predicting it. But, there are all of these, I think, considerations that, that would you would, you would have to do. But, again, we’re not starting from scratch. There’s a whole, you know, experience that we can draw from, in how we in how we do these things. Got it. I wanna just open it up. Does anyone wanna comment, Shlomi and Alvin? Maybe, again, the new technology, puts in in front of us, new challenges, and even all channel challenges needs to be addressed differently. So when you’re talking about how do we evaluate accuracy, there’s one part of it. Okay. So how do I evaluate accuracy, but also how do I influence the accuracy? Eventually, in classical machine learning, you can retrain a model. Usually, in generative AI, big models that are currently released, the way to influence the model is is less direct, and the way to validate the results is different. And, we need to get used to those, systems. You can ask Charge Ept, and it might produce, in a summary, let’s take the summary use case, something that is completely relevant to the communication that happened there. How do you validate it? You cannot just retrain the model or influence it directly. You need to put in place all kinds of guards and controls, ask it multiple questions, use multiple agents like multiple ways of framing the question to the model. Validate it against external data sources. I’ll give you a small example of it. Let’s say that I give a large Gen AI model communication and ask it to find a I curate the question well and I ask you to find a financial risk in it and it answers me. Now I need to reassure myself that the answer is correct. It is a huge model with a lot of knowledge. Ninety nine percent of the time it is correct, but it also sometimes even invents things, things that weren’t our concern in the past at all. I need to to find metrics for that, and those metrics are currently being developed. So we can use different actors to ask, the model. We can ask it, in a more severe manner. Please make sure that you’re guarded. We, that you’re cautious in your answer. We can ask multiple models. We need to to use multiple layers of technologies in order to validate the question, and we can also confront it with additional information from outside. For example, if I get an answer, there is insider information here. Maybe I put in front of it the definition of inside of information and validate. Are you sure that this is matching your original answer? Right. Advanced sharing models are very good in, doing what we call chain of thoughts, meaning rethinking their, entire decision process and being able to reproduce an answer. And there are multi multiple advanced techniques in the data science world on how to operate this correctly. These are the kinds of controls that you need to put in place today in these kinds of technology. Right. And so you say you, are you putting the onus on the vendor, the onus on the firm, or the you know, like, like, who’s putting these controls? First of all, when every vendor releases a product like that, it’s on on their responsibility as well. Of course, you need to make sure that, you put in place all of these controls. You cannot offer a product that, will create a hallucination for for the customer. In terms of it is, eventually, our customers are, a big financial organizations, and they’re the ones that are exposed to regulations. So they need to make sure that they have in place all relevant documentations, metrics, evaluations, testings, ongoing monitoring that they can show the eventual the regular eventually the regulator that these models are up to their needs. Maybe one more, thing to say about, concerns that currently raise, it’s not only around hallucination and accuracy of the models, again, which are more difficult, to measure. It’s also and we see it in regulations. FEMA started talking about it, the new AI act that is coming in the upcoming two years is also discussing it. The fact that we have a far more robust technology, including all trends of regulations in the, all around the world is is scaring everybody. And the the scare is where will it meet me? How much, more danger can it put me in? I have more robust machines, more robust technologies. I need to put more guards in place. And additional concerns, not only explainability, and the, accuracy in place, We see that now, for example, in the new AI act, risk, of AI is being measured not only by the robustness of the model, how large is your model, but also on the use case. What are you trying to do? Will it affect, union workplace? Will it affect their privacy? All of these things and And that way we that’s why I think we’ve seen things like profiles come into questions too. Right? So I think a question we’ve yielded for a long time is can we develop a risk profile? Who’s gonna how do we know the LIBOR before the LIBOR and that kind of thing. Right? And so much Exactly. And they and they Yeah. And the EUAII specifically calls out risk profiles. Right? And they’re still refining that that that that that regulation is still being refined. Right? But there are some use cases that may be harder to get across because of regulation and some that aren’t. Now I think a few of the things you addressed, you talked talked about sort of building prompts into LLMs as a way to refine them. Right? I think, validation criteria people use. I think, Alvin, you and I, when we were talking before this, where you you were talking about other controls people can put into, their models as well. So do you wanna talk about some of those rack controls and and and other things discussed? Yeah. Absolutely. And I was gonna say maybe the approach I’ll provide, is gonna be something that’s, a little a little less technical and maybe even a little less rigorous, but it’s kind of a good mental model to think about. Right? Before I kinda jump down there, just for those on the call who, aren’t aware of hallucinations, hallucinations basically is when the large language model just makes something up. Right? If it doesn’t know the answer, it’ll make something up. And for obvious reasons, that’s a big concern within financial services, industry, right, from kind of a compliance and regulatory perspective. Now there’s a couple of different ways to address it. The analogy I’ve heard, which is a good one, right, is treat that LOM as a person. Right? And what do I mean by that? So if you treat that LOM as a person, there’s a couple steps you can take. Number one, just let the l m know that the answer I don’t know is an acceptable answer. Right? So if you just simply just tell the LOM, say, hey. Look. If you don’t know the answer, just say I don’t know instead of making something up. Right? So that’s number one. Number two is this RAG approach that, Alex talked about. Right? So RAG stands for retrieval augmented generation. And, basically, what RAG is, it’s an open book test for the large language model. Right? You provide a data source. You provide a data repository for the large language models to reach into to find the answer to your question. Right? Now one of the customers, has done this, right, where they put kind of their trading rule book, right, and hooked it up to an LLM so that their traders can ask questions to say things like, hey. What are queue orders, or what’s the regulation around x, y, and z? Right? And then what that large language model will do is it’ll actually just reach back into that data repository repository is a very, very light lift. Right? For example, if there’s that data repository is a very, very light lift. Right? For example, if there’s a new order type that comes out or if a regulation changes, great. Upload the most recent document. Right? And now the large language model has access to the most recent information and can provide you the most up to date, most accurate information. You don’t have to retrain a model or anything like that. Right? The second benefit is the lineage. Right? Which means that when the large language model gives you an answer, right, and just like as a person, right, when the when when you ask a person a question and they give you an answer, if you don’t like the answer they gave you, you ask them. You say, hey. Where did you get this information from? Right? And that associate, that colleague of yours is gonna gonna go back to say, well, I got the answer from this document. Right? You should expect the same from your large language model. Right? To say, hey. If the large language model gives me an answer, I don’t really believe it or I have some questions around it. Hey. Where’d you get that answer from? Right? And using this RAG architecture, it can do that. Right? It can point you to say, hey. I got the answer from this document. And then kinda to show me, mention this. Right? Is it also then provides you that chain of thought or that step by step process in terms of how that large language model took that information, synthesized it, and how it arrived at its conclusion. Right? So, again, I think mental model wise, treat that large language model as a human. Right? So if you don’t believe the answer, ask it to tell you where it got the information from, right, and how it, did its thinking in terms of how it arrived at the conclusion. Right. And I think the interesting thing we’re all kind of harping on, right, is a change in mindset where we’re going to a world in which the efficacy of these models, the efficiency the efficiency as we understand, right, but the efficacy of them is being measured by their outputs. Right? The things about their lineage, right, the datasets they were trained on and whatnot that are much harder to understand. So in Laura’s world, not that long ago, he could say, you know, these are the, you know, twenty thousand emails I used to train this particular model, and these were the sentences, and someone could hold them. And I think it makes, you know, intuitive sense that you can hold something and look at it and feel like you understand, like, well, that’s how it became this. Right? And I think the leap that’s hard for people is, well, if I don’t know, you know, where it came from, how can I be comfortable with what it’s telling me? And I guess what we’re all seeing in our different ways is is that there are ways to measure that, and there are controls you can put in place. And so that the validation people need to do now to become comfortable with the tools that they’re you know, whatever it is they wanna take on or try or experiment with or ultimately deploy, They need controls in place to be able to measure those outputs at various levels. The lineage of the information, what prompts are baked into it as a way to validate those answers, and then ultimately, the actual merits of what comes out on the other end. Does it actually meet what it was asked to do? Right? Does that feel right? I think one interesting thing go ahead, Laura. No. No. Just to add, I I really like, Alvin’s analogy of the, treated like a human. And the question about that, I was thinking about this the other day. Again, in in your, you know, more, traditional ML, machine learning, you have scores. Right? You have likelihoods. How close is it to what you trained it on? So in base of that, you can say, okay. I’m gonna alert on the highest scoring ones. I think, you know, folks are familiar with that use case. That’s like a measure of the confidence of the of the prediction. If you ask that LLM as a person, how confident are you in your answer? Do we have a metric for that? I don’t know, Shlomit, if you have that. Yeah. So I don’t know if confidence you know, eventually, I think that the the first thing that was discovered about Chatubit and the jokes and the, you know, youth networks. Well, we’re all around that Chatupiti will tell you everything, Chatupiti, and all Gen AI models after that with great confidence. It is sure of its answer. I think that the new advancements, that, were done in the past year is, again, through the chain of thought process, its ability to reflect and to think back given additional information or given that you are presenting it the question with different aspects, different angles or presenting it with new information as Alvin is saying. Okay. Go verify it for me. I can ask it where did you retrieve the document from to start with, but I can also ask it go look inside the the regulations now and, make sure that you made the right decision according to the definition there, not only according to your own knowledge. So there are a lot of techniques that allows us to increase our confidence in in the result. I don’t know if it is, exactly the same as receiving a confidence score from the model itself. It’s just making it answer it again, asking it in a different question, producing an agent that is a critic to it, adding additional agents that, force it to look in additional information. Right. Etcetera etcetera. And I think I I would add one other element, which is a really basic one, right, which is just QA. Right? I think and and I think taking yourself on that journey, I think people also think of this journey as kind of binary. You you’re switching, you know, you’re you’re hitting the switch, and now everything looks like this all of a sudden. Right? But I think, firms should, you know, they should walk there. Right? Because you’re gonna get there. Right? You know, if you’ve pointed yourself in that direction, I think, Laura, one thing interesting you said when when we were kind of talking in advance of this discussion is, like, these are ultimately all risk based decisions. Right? So if you could take a lexicon, right, I may look for the word rumor, and I know that I miss every time rumor was misspelled or maybe I spelled it in an American way and not a European way. I’m missing a lot, but I know what I’m getting. Right? And so that’s that’s why I can go to sleep because that’s it. Right? And I can add words and that kinda thing. It makes sense. Right? Right. Then you take the the supervised model. Right? Then we’d have a threshold, you know, and I would look through those results. Right? And there are always a few sentences below the threshold that were kind of like, well, that kind of looks like what I want it to be, but it’s sort of you know, it’s clumped in with so much other stuff that I don’t wanna alert on that the risk based decision I’m making is to look at everything above this threshold. Right? We’re not asking people to make any different decision. At the end of the day, how are you comfortable monitoring for what you think you may have missed, and how confident are you or how comfortable are you around the controls you have to validate what you wanna find. Right? And it’s really just making the switch to how we develop those controls, I think. And we’ve at this point, we’ve listed a bunch, right, from, you know, prompts to to rag, and I’m just saying basic QA right across what you’re not sure, you know, whatever corpus you’re not sure of as a way to, get comfortable. We’re gonna segue into best practices, how people vet this kind of stuff, and that kind of thing. But before we do that, we do have a question, and I wanted to just kind of attach that. So it’s sort of, in your view, your collective ChatGPT versus Copilot three sixty five in day to day compliance operations? We have the option to use both, but it appears that the majority of team members are struggling with use cases. If you could provide some specific examples, it would be greatly appreciated. So let’s start off let’s help this, let’s help this caller out and, just I think of some more day to day uses of GPTs, but I think talking about use cases will help us kind of segue into the conversation I wanna have just around, you know, use cases and and AI more generally. So any thoughts on, what are some good use cases for Copilot or or ChatGPT? I’m I’m happy to start, Alex. And and I I kinda gave two. Right? You know, one was around, like, if you just want a simple kind of q and a chatbot, versus against kind of rules and regulations. Right? So you can kinda dump all the rules and regulations, especially if you’re a global bank, like, you know, where where the lawyer’s working. Right? You probably, interact with hundreds of regulators, right, all with different, rules, across the regions. Right? And you may have a a question around, hey. What’s the the trading rule in Zimbabwe? Right? So if you kinda just dump all of that into a data store, right, hook up an LLM, right, you can create a q and a chatbot, to get answers, right, in natural language right across that. Now I am shorthand. I’m I am hand waving a little bit. It’s not quite that easy, and and, you know, there’s a lot of complexity behind how you set up a data store and a betting model, etcetera, etcetera. Right? But high level, right, that’s kinda one use case you can use, you know, in terms of, generative AI. Right? The other is around, alert disposition, right, or alert resolution. Right? So, again, if you have an alert that gets generated today for a front run the alert, right, or an insider trading alert, you need to collect data from multiple different sources. Right? You may have to look up the company’s filings. You may have to look up, old investigations, etcetera. Right? You can leverage that GenAI to, again, grab all that data for you, summarize it. Right? Right? And then it it reduces kind of that manual effort, you need to do that. No. It’s these two. And Go ahead. Sorry. Just one one last these are two use cases, right, that are, actually in production, right, by, financial institutions today. So they’re not I’m not making these up. These are, you know, real, production use cases. The the only other thing I’d I’d add there is, for coding assistance, right, which is which can help really in the development platform and time to market. I mean, that that can collaborative coding as, with the use of a Copilot. I mean, that that is that is a known use case, and that is known to be productive. Right. Now if I could summarize, pun intended, Maybe maybe I can add one more Yeah. Please. In general, I think that all organizations should, have already a connection to ChargebeeT or Copilot or both of them. It’s just a matter of convenience. What’s security wise and, Chargebee is now allow OpenAI allowing connection to Chargebee as well. But I think that the main difference and the main advances that will be in this, industry today is that when you have a connection to such a large model, it all depends on the fact that you have a question and you win you want to have a direct answer on it. Then it is great to direct this question to the model and it will give you great answers. It’s like it used to be looking in Google, now we’re asking Chargepitti. But the future is to have a platforms or products that do not require you to think of the question or to, initiate the to to to initiate the understanding that there is something missing here. But the platform itself will already and, I think that we touched upon it a little bit, before here. The platform itself will already initiate and collect information for you and put it in front of you, collect risks for you, and make decisions for you even before you know that you have a question raising. So when you already know that, you have data missing or a question that you need to answer or you don’t really know how to implement something in code, great. Chargeppt is the way to go. But in terms of incorporating it in products, there is also an additional leap that needs to be done, And how is the product leveraging it be without the need of you actually initiating the question, but just putting it in front of your eyes. But but I think But anyways, alerting is being done today. For sure. But I think, like, so we could segue, I think, into best practices and use case. I think the point you made, Alvin, around regulations. Right? I think to to answer the question, I think if they can narrow the focus of what they, you know, of the corpus they wanna ask across for specific use cases instead of saying look across the whole bank. Right? Load all your regulations in and see what performs best. Right? Load all your policies in and see who give you know, what gives you the policy you answer out of those questions. Right? And then looking ahead, some of Shlomit’s considerations, okay, of of how this can be, you know, refined and improved from here. Right? The place we’re at now is one where for the GPTs in day to day use, people are gonna the point is to ask you questions is kind of where we’re at. And where we’re looking to get is where they where these machines can start to do more reasoning and help us get better, you know, better insights across that kind of thing. Now speaking of use cases, right, I think it’s a useful it’s useful to sort of talk about this first because I think we we we tend to think of the whole first. Right? We tend to think, you know, from a surveillance perspective. Right? You know, I I I, you know, it’s it’s again, you know, show me Libor, show me show me whatever show me and often something show me something that’s already happened. Right? Or help me see the future. Right? And it’s not a useful approach to this technology. Right? Because it’s just too muddled. Right? A way to get to that answer. Right? So I think let’s talk about how we focus our use cases in a meaningful way so people can start to leverage, think about, and experiment with AI and, sort of open forum who who wants to kinda pick it up? I can probably start here, as well. Right? I I think look. The considerations going in, I think take your usual approach around use case, which is to have a well defined use case, or well defined problem that you’re looking to solve. Right? And then and then seeing which technology helps solve that. Right? Like, again, don’t use Gen AI essay hammer. Right? And and everything will look like a nail. Right? But, you know, sometimes traditional machine learning will help solve that. Sometimes, you know, basic analytics will help solve it. Sometimes you need, generative AI. Right? But I think the the biggest shift that I’ve seen over the last, say, twelve to eighteen months, right, is eighteen months ago, people want to boil the ocean. Right? People kinda say, hey. Gender of AI, it’s a, you know, it’s a model that can do multiple things. Right? And and we can solve all of these different problems. Now what what we’re seeing is people are more focused around the return on investment, right, to say, okay. If I’m gonna apply GenAI to this use case, what’s my ROI on it? Right? And I think that’s the right approach to take. All the examples I gave, right, have a well defined ROI. Right? It’s saving people x number of time. Right? It’s reducing a manual effort by thirty percent. Right? Call center use case is another one. Right? If you’re able to reduce the the call center time from twelve minutes down to four minutes, right, you know exactly how much money you’ve saved. Right? And you know exactly how much that return on investment is gonna be. Right. And I think return on investment also means, like, I think people don’t fully appreciate how much it costs to crunch all this data, right, through one of these machines. Right? So we have this consumer experience of just asking questions. We get an answer. It sort of feels like that was easy. But to to do some of these things at scale across all of a bank’s comms or or or or what have you is really expensive. So unless the output merits that money, it’s really hard to, you know, to to justify taking that big a leap. Right? And then that go ahead. That that’s a great point, Alex. So so how do you take that so what’s the implication of that? We’ve we’ve we have a bunch of use cases. Every function in, in a large organization, a large bank will can’t wait to get their hands on on and experiment on this. But what we found useful is, do we have, like, a a governing body, like a like a process centralized process that looks, it knows all of these potential use cases, evaluates them consistently, like, kinda assesses them through different dimensions and really asks to prove out, okay, which ones from both a risk perspective, a, a productivity perspective, a potential success perspective, which ones of these do we start with. Right? So that you you have a a systematic and and and transparent process for, for governing and and proceeding along your Gen AI journey. I think that’s useful. I mean, we have a a center of excellence, for example, that, even before GenAI has, principles for ethical use of AI. If you have that and you apply that through the that same lens of human centricity, fairness, privacy, evolvability, because, you know, in my team, I don’t certainly don’t want to be restricted to the knowledge of my of of surveillance. I mean, there are other, folks who’ve been using this in in other domains that can potentially inform the way I use it. So if you have that, like, enterprise wide, body of knowledge and governance, I think that that really helps. Can I ask you, does does that ever hobble experimentation, though? Like, in as much as, you know, so much of this stuff is new, you might wanna just figure out what it can and can’t do, but you have to funnel it through, like, a a specific use case in order to, like, like, what’s that dance like? The the the the spirit of the data scientist is just to, you know, jump into it. But but the risks are, you know, it’s reasonable to be, aware of the risk and to address them. And and as long as it communicated so to answer your question, it hasn’t. Of course, it could have been faster if you didn’t have that, but there’s a cost to not having appropriate governance. I mean, regulation is existing and forthcoming. You wanna make sure that there’s, you know, genuine cost of these things if you’re not, prudent in your adoption. Alright. Shalomie, do you have something to I saw you kinda nod in there. Did you have something to to add there? So I think that in terms of, the evolvement of the the use of GenAI, first, again, there was, everybody were, amazed by its capabilities and then came fear and the regulation started to acknowledge the fact that it is going to be here and how to handle it. And, it is true that now the value is, the focus. How do I get, real value from this, Gen AI? And again, the first use cases are trivial in terms of let’s ask it, let’s have it, summarized, let’s have it, give me answers, but the more challenging ones will be how will it make decisions for me? How will it do the work behind the scenes for me? This is where the real value will, come in place, not only in terms of doing better what, we are doing so far, but also in terms of looking at the problem from a different perspective. I heard you, Alex, once suggesting instead of looking at alerts, for example, in the communications of Ellen’s, let’s look at risks. Now we know that we can collapse the information across alerts. Why are we looking at communication by communication? Why can’t we look at at it from a different perspective? This is, the change of the world that will occur in terms of, the new technology, but the correct judgment of it eventually will be how much value does it give me. Right. Alvin, over to you, and then I’ll just add a few last thoughts. I have nothing more to add to what Sonia and Laura said. So Okay. Great. Well, you go. Your contribution has been great, so all good. So I think look. I think if we’ve covered a few things today, I think we’ve looked to refine the value, right, so that people are thinking more squarely through what the value is. Right? Everyone understands that that, you know, that these capabilities are there, but but but what’s the actual value? I think we’ve also looked to emphasize that the controls exist and are continuing to mature around how we measure that value. And I think firms need to lean into and vendors, right, need to lean into establishing what those controls are, showing them around, and iterating on them in a way that people can become comfortable using these tools. And then finally, on the other side, sort of as firms look to bring those on or think about them, what are the best use cases to deal with? Right? And I think educating themselves, working in partnership, you know, with whoever has these capabilities, and ultimately, not being afraid to experiment. Right? As long as they can be, you know, they can document kind of what it is they did. Anyways, I feel like we could go on and on and on, but I’m really grateful for this panel. I’m really grateful for this discussion. We covered a ton. So for everyone listening, thank you for listening, and have an amazing day.