Help the agent know what is important, what is not —
and prevent it from making silent decisions.

Near-verbatim. Backchannels, mic small talk and the interviewer’s opening briefing about the series have been trimmed; no paraphrase, no speaker merging.

~00:04What Nazmi is

NICOLASYou've been building AI for a long time. You're currently solo building, you've been a founder in a couple of ventures, and you currently operate under Nazmi, helping — how do you put it — production-ready AI agents, client-ready AI agents. In your own words, how would you describe it?

ISAACSo Nazmi is more a consulting company delivering senior-level AI services. It can be diverse — from MLOps, deploying machine learning and AI models, to developing agentic systems, RAGs, knowledge graphs, memory systems, depending on the customer use case. It's more the consulting side of building these solutions. In addition to that, I've been doing side open-source projects. Lerim was one, which was more on memory management — basically a background agent which does memory management for you. These days we see Claude and Codex, for example, have added this memory system as well.

ISAACIn addition to that, because of the PhD in multi-agent reinforcement learning and decision-making I had, these days we see those coming into LLMs as well. So that's another project I'm working on — post-training and training small models for a specific task, for example customer support or database interaction. For that kind of task you don't need a very general, expensive model; this kind of specifically trained model can do the job. I have published a few blog posts on those. For example, using a 9B model I could get GPT-5.5-level performance on some databases. Or an under-one-billion-parameter model — Qwen 0.8 — was able to get to almost the level of a Qwen 35B model, or 78% of GPT-5.5.

ISAACThis kind of side exploration comes from the interest between product and research. I'm trying to take the research into the product, and this is what I had with clients as well. So it's not only engineering — because the field is on the edge and everything is changing, you need to be up to date, read a lot. It's kind of needed to know what is going on under the hood, how these models are working, to be able to build better systems.

~00:07Why this work

NICOLASBefore going into a deep dive — out of everything you can do in the world, why are you trying to learn and operate in this space with AI models? Why does this tickle your interest? You could be a musician, you could be a plumber.

ISAACSo I think it's a combination of several motivations and feelings. Building is one. Learning is something else — my whole life I've been learning new things, doing new things. This AI space I feel is something that can satisfy these feelings: of learning, of being a creator.

~00:09The filtering machine

NICOLASOn the learning side of things — since this AI space is moving so fast, what are the channels you follow yourself? How do you keep up to date with everything that is happening?

ISAACOne is I spend a lot of time on Twitter — that one you cannot remove. The other one is I have a Telegram channel for myself, which has around 400 people subscribed as well. I send every link that I find — from YouTube, Twitter, LinkedIn, papers, blog posts, any link basically — and I have an agent that summarizes those there and generates one audio clip as well, one to two minutes. Then daily I just read those. So this is one filtering layer, kind of me or the agent selecting the top ones, and I then read and select one or two and post on LinkedIn for others to share as well.

ISAACTwitter is super up to date, and after a while, once you follow the correct people and with the interactions you have — likes, shares — the recommendation system becomes super helpful to filter the things that you want. So Twitter I would say is the first source, and then my Telegram channel is somewhere I select the top ones and spend more time on them.

NICOLASAll of the links you share — do you store them in some sort of memory, or your own RAG, for implementation reasons? Something you can refer to, not just content-wise but actually for building?

ISAACSo Telegram history is one, but my agent has a SQL DB. It basically reads the Telegram channel, and I send links to Telegram, and then the agent summarizes them. So everything is in a SQL DB in the backend.

NICOLASVery efficient. Do you share that memory system with others, or with the followers?

ISAACThat's on my Mac locally, but the summaries and everything are in the channel and open. So links and summaries, audio, voices. Each post has two replies: one is the text summary and one is the audio version of that text, using open-source text-to-speech models.

NICOLASVery nice setup. Still on the topic of learning — three people one should follow, in your mind. Three people you follow where what they share is super valuable.

ISAACMostly company accounts, actually. One is Prime Intellect, which does reinforcement learning — my favourite topic. Then the famous people: Andrej Karpathy. Or Claude's accounts. I'm not into hype. Sometimes some topics get hyped, especially these days — loop engineering, graph engineering, etcetera — but I'm a bit resistant about that. So mostly these famous companies, and because of the recommendations it's more about topics, and then suddenly different people come up. Basically on Twitter you have one part which is the people that you follow, and one part — the “for you” — which finds different people related to those topics.

~00:15Lerim, and the hardest part of memory

NICOLASI read the Medium blog you wrote in April about the system you mentioned — Lerim, the open-source project that helps with memory efficiency. On that topic, what is the hardest part in implementing a fully autonomous system that actually uses the product data?

ISAACSo basically the idea is that there are a lot of data — chat traces, agent traces — and we want to build a kind of context layer on top of this, to filter and basically extract the decisions that have been made, the different signals, from those traces. This can be applied to data as well: you have Slack, Teams, etcetera, and you want to extract some signal and then use it.

ISAACAnd the hardest part, especially for memory, was to find the balance — not to memorize too many details, but also not to fail to memorize those that matter. So the harness can be important: different hierarchies that you have. You extract something, then you filter those, another layer, and you do for example another dimension over time — to merge the similar ones, forget the ones that are not used. This kind of stuff that, for example, Claude calls dreaming. But this is similar to the human mind: you have a lot of information, some of it you keep all the time in mind — for example one agent can be there and summarize what you have been doing in the last week or months. Some facts are longer term — six months, seven months, one year — and need to be recalled and reused in the agent. Some information just needs to be in a database, and the agents, based on the need, will retrieve and use it.

ISAACSo all of these components and harnesses, how to work together, which model is good enough and not expensive — because this is a huge input. In this one I was using a Minimax model. First I was using a completely agentic system. Then I tried to build the harness, which is intelligent enough to remove the context that it doesn't need from its context window. And in the end, after all of this, I came to the conclusion that a simple workflow can be enough — several components, LLM calls, okay do this, then do this. Much more predictable, easier to build, and you can use smaller models.

ISAACFor example, one of the use cases of training a model would be, in this case, to train one model just to extract these facts for me. So basically you use a super powerful model — you can use Opus here — just to record the decisions, and then fine-tune a small model on this to mimic the behaviour. This is kind of knowledge distillation, the kind of fine-tuning or post-training that people do.

~00:19Silent decisions

NICOLASThank you for elaborating on that. In the way you've been using AI agents and the way you like to work with them — where do you want to put yourself as a human in the loop? Where do you think the human is best positioned in the work you do?

ISAACI think one of the things that I try to do is to basically help the agent know what is important, what is not — to try to prevent it from making silent decisions. Splitting the work.

NICOLASWhat is a silent decision?

ISAACFor example, when I ask an AI agent: go and train a 9B model on this task, this is the benchmark downloaded from Hugging Face, then train a model. It does the training for you. And at the same time, when I ask it — okay, we want to do this, let's first explore the data, we download the data, we say okay we want to train this model on this, let's for example filter the data — so this is just pointing that okay, this is 5,000 data points.

ISAACI was doing this in the morning, for example, and then I checked the data, and for example 1,000 data points from the dataset were kind of not having some information that the agent could use. I was seeing that it is failing. So I started to break down these steps. Then the agent does it, but I confirm the results. Then I say okay, data seems to be good, let's go and do this. So basically you spend more time, but in smaller steps like this I was getting better results.

ISAACBut there is an issue over time: my trust in one model goes up, and then the performance goes down. Then again I start to reset myself, my expectations, basically because of the media hype. People are like, okay, we put one agent, it works for 10 hours and it gives us this. Maybe that works, but you need to build that system — okay, these are the criteria for the data, these are the criteria for the model. So whatever I have to do — this is in the startup space, they say okay, do the hard work first yourself, which is not scalable. So basically you do all of this step by step, you convert it into system steps. Then maybe an AI agent can do a better job, versus the one where you say okay, go and do the training and make the decisions. So all of these in-between decisions are basically the silent decisions.

NICOLASBut normally you would have the agent make some sort of plan of what it does. Are you saying that even if you make a plan — and even if you just one-shot a badly instructed agent — you still don't necessarily know where it might fail? So you're there as a human, structuring it.

ISAACYeah, exactly. The planning — basically that system that I mean can be some sort of plan steps. But for example, sometimes these in-between steps need me as a human to be there, judge, and then continue. For example the dataset case: in the beginning it can say okay, this is the number of data points, I do this, then I do this, then I do this. I say yeah, that's good, but after step one let me check the results. Maybe there are some charts, maybe I have some insights from someone somewhere else, and then I say that okay, this part doesn't seem to be good, let's do this. Or let's just go and research and find literature on this — maybe there is some insight. Because of the limited capability, or basically the time that the model has spent, sometimes just giving it a word can change the direction.

ISAACBut definitely, if I know many of these beforehand and can make a proper plan, I think that would also work. I'm trying to get to that plan, but do the whole process if it's repeatable work several times.

~00:23Build or buy

NICOLASAs a person who can build his own agents for any given task — when do you decide to actually build an agent, and do you go and check if somebody has already solved this, or plug-and-play solutions? What is your default mode, and where do you go to search for solutions?

ISAACBecause of the research background, I spend not a lot of time but some time on this — review background literature, what others have done. And then if nothing is basically solving my issue, I try to build it myself. Sometimes maybe startup ideas come out of it: I spend some time and then leave it.

ISAACYesterday, for example, I was listening to an audio summary of a book, and then I felt okay, it's a Blinkist — 20, 30 minutes, not too much detail. I'm looking for something which is, I don't know, two hours of summary of this book which is 10 hours on Audible. I searched and I found, for example, Shortform, but I thought okay, it's two hours, it's 25 per month — can I have a ChatGPT Pro subscription, can I somehow just get the book, summarize it, and then use text-to-speech to build the audio? So these kinds of approaches: looking for something, then if not available, build it myself.

ISAACFor example this Chrome extension to summarize the page — I have built an open source one. Or speech-to-text, text-to-speech: one of the issues I had was okay, agents give me text, I'm lazy to read all the text. There is one open-source project which you select the text and it does text-to-speech for you. So if not available, I would try to build it, depending on the time and importance.

NICOLASThat's interesting. I think ElevenReader would likely be the closest to actually implementing something like this — where you would have books as PDFs or whatever, and then you tell it what sort of summary, what you want to highlight, how long it should be, and it likely returns something like that.

ISAACElevenLabs is one option, yeah.

NICOLASIn terms of podcast stuff, I really love how NotebookLM generates the podcast — they're surprisingly entertaining when it comes to digesting fairly long content into something easier.

~00:28What doesn't compress

NICOLASWe're five minutes left in the conversation. Is there anything you would have hoped I would have asked, or expected I would have asked but didn't, and you'd like to highlight or comment on?

ISAACI think one important thing which we feel the effect of these days as well is about the effect of AI. We see a lot of layoffs. We see a lot of uncertainties in people, especially in tech — developers, for example, think that AI will replace them. That would be an interesting topic. I don't have any answer for it, but I mean, related to the topic that we talked about: when the human judgment comes in, is the expertise still needed, when can it be helpful?

ISAACSo I still think that companies can be smaller, for sure, but human judgment is where it can basically make those differences. Many people can run Claude Code and tell it to do, I don't know, even the most difficult things that we see these days — okay, go and train a GPT-5, for example. But those expertise details: how to collect the data, clean the data, how to train, what is basically the role of this component, how to make it better. Maybe AI gets there, but I don't know we can do much more without it.

NICOLASI think there are two elements to that. One is the human can make the judgment shortcuts — knowing when you need a big model to solve a problem or not. You can be super cost-efficient because you don't put an agent on this. It's like you don't ask an AI how many R's there are in strawberry — that's not a problem you want AI to stall on. It's like bringing a bazooka to a knife fight; you don't need it.

ISAACYeah, I like that.

NICOLASAnd the other part is there might actually be value in the human accountability. Even if AI can do everything, you will want to put a human on this, because you can't hold the AI accountable — and if the AI is doing too much, it's really hard to apply accountability. So there might be value even if the human isn't necessarily adding value in the narrow sense: in our business it adds value to have a human here, and we won't substitute for that, and we'll take the human flaws over the non-human flaws.

ISAACExactly. No, of course.

NICOLASI'll share you a link to this video — Claude made a video with a creator who explained why people still watch chess even though AI solved the chess problem a long time ago. That's the other part: when it's not accountability, for some reason humans are still fiddling around with this, and then there are other humans wanting to watch the one person fiddling around with it, even though it could be solved.

ISAACThat's interesting. Yeah, that's true. For sure.

NICOLASThank you — happy to follow up. And it sounds like your Telegram channel has a lot of good stuff. I would be delighted to learn from you, because I already enjoyed the content I dwelled into.

ISAACYeah, I can share as well. Thank you.