Hello, I'm Mat Ryer, and welcome to Grafana's Big Tent, the podcast all about the people, community, tools, and tech around observability. I'm here with a friend of mine, Tom Wilkie. Hi, Tom. How's it going? How are you?

Hi there, Mat. I'm very well. I think congratulations are in order, I hear.

Oh, is it? Go on then.

Didn't you recently change roles at Grafana Labs?

Yeah, I did indeed. I'm now going to be the senior director for AI over here at Grafana Labs.

Congratulations!

I was young and I needed the money. We are building some very cool things. We have met Eric in person, so we know he really exists. And we are building a lot of those cool things on top of Claude, one of our favorite models in the world.

Speaking of which, not that he's a model. Or are you, Eric? Have you done any modeling?

Yeah, and he is good looking.

We've got Eric Burns joining us from Anthropic. Eric, you're a field executive architect. What's that?

I think at least the way that I think of it for myself is that I'm building bridges of understanding between the tech execs in the room and the business execs in the room when we reach the moment of making a significant strategic decision.

So, you know, I think of it kind of as there are two local maxima of making strategic decisions about AI. One of them is, who should be in the short list and who are we going to pilot? And the other is, as we start to converge on a vendor and start to standardize, who should it be? In those moments, it's helpful to have a shared narrative and some kind of shared foundation and to occasionally have some help to interpret the language that each group is speaking in, in a way that converges in strategy. That's the top line.

I love that.

But in a lot of ways, I'm a curator of vibes. When I'm getting together with exec teams of partner companies, the vibe should be good. And they should think, "Well, we've got to work with these guys."

Yeah, indeed. As Mat said, we're a pretty big fan of Anthropic models. We do use models from other vendors. I've heard they exist. But I think one of the things that attracted us to the Anthropic models was the availability. We run across all the major cloud providers, across Microsoft and Amazon and Google and so on, and the fact that you can get your models in region across all of them was a big win for us. How did that come about, or why is that Anthropic's strategy versus, I know Anthropic and OpenAI and Google have very different strategies for model availability.

Yeah. I've only been here coming up on two years, which makes me near old guard at Anthropic, but not there for the early days in the origin story. It strikes me as kind of a classic second-mover situation where there was an early breakaway leader and they had kind of a slightly more vertically integrated, or at least single-partner, strategy. When you're trying to get your feet under you as a startup, you try to figure out where your opportunities are and you maximize what you can do within the opportunity space.

So I did a startup before this. One of my favorite quotes from that time was something like, "Great architecture is all in the constraints." If you're a second-mover lab, one of your constraints is that you need to line up a hosting partner and you need to tell a good story for that. I think it's a very significant maturity transition for a company to be able to work with many different partners and many different platforms. It's kind of ripping the band-aid at a technical level that you can put in all of this work before the first dollar of partner revenue arrives.

That's a substantial opportunity cost, and it might fail, and you might be left without a good result. But I think one of the things that makes Anthropic is the decision to pay that cost early and to generalize and be able to work across clouds. You can see that now in being the first model provider to reach all three major hyperscaler platforms. I think one of our core competencies is making our models run on a diverse set of hardware.

They'll use that at some point to embarrass me, I'm sure. Go nuts. Don't worry about the awkward pauses. They'll edit those out. Otherwise the pattern is I tend to just ask 50 questions and Mat just sits there watching, so I'm going to awkward pause and let's see if Mat has anything he wants to add, or they'll leave that in and it'll be really funny for the audience.

Yeah, I guess to kind of give a little longer intro. So you said, with your story with Anthropic, I was interested in when did you actually realize that, oh, hang on a minute, this LLM is actually working? This is something different.

Yeah. Studying computer science, I started out at Carnegie Mellon, did a bunch of research work, and kind of fell into digital libraries and information retrieval. Most of my career before the AI era has been traditional IR, which these modern systems make a mockery of. But it was being able to organize lots of information and retrieve it with high recall and have really strong relevance, and just all the stuff that makes a good search engine.

I worked on this company called Panopto, which in the UK at least is very popular in higher ed. We were accumulating lots and lots of basically transcripts of primarily higher education lectures, and some large enterprise. It just seemed like this enormous corpus of information. Our claim to fame as that company was, we're going to search a multi-hundred-thousand-hour video archive.

We kept having these opportunities to integrate. I watched large language models progress as a technology category in slow motion. At the time, I was the CEO of the company, and so I was always on the lookout for technologies that would be complementary to our core value prop, which is, you have lots and lots of video. It's hard to produce, hard to store, hard to manage, and hard to get value out of. We will light it up and make it searchable and manage access controls.

At a certain point, being able to actually understand the content of a transcript as opposed to summarize it in a kind of clunky functional approach, or... There were clearly ways that we could get into some kind of synthesis, but it just wasn't ready yet. Over the course of about 15 years, built the company up, eventually found an exit, and I had a chance to kind of look around at the landscape and figure out what was going to be interesting next.

I could just feel that AI was going to be the next thing, but it didn't have that kind of killer app. I'd already kind of become a believer from a lot of machine vision stuff got really good really fast. A lot of speech recognition stuff got really good. Of course, AlphaGo. You sort of see it building. And then, this is the most cliche answer, but the first time I used GPT-3, I got shivers, and I kind of had this moment, as I'm sure a lot of people did, of like, this is it. This is what's going on.

My angle on it was very different because my first reaction was, holy cow, this is the machine that can make sense of lots of data and can actually perform the synthesis step that's kind of like the keystone of the arch. If you make it easy for people to make lots of content, and then you aggregate all that content, and you light it up, and you give it a baseline of searchability, the missing link is being able to bring it down to a manageable level and to have
the bandwidth of what you're delivering to the user match their scenario. It was clearly the missing black box in that equation. That was the moment where a light bulb went on for me, and this kind of gets into going to Anthropic. I believed there was an imminent convergence between traditional information retrieval and using large language models to process and synthesize the info. I think that's what we're seeing unfold at a large scale today.

Yeah, I had that same experience of just seeing it actually doing things and surprising us. When we built the Assistant project, we have a Grafana Assistant, which is our built-in chat. It's like having a copilot with you while you're using Grafana.

Don't say copilot.

Not a copilot.

Yeah, yeah. I'm playing other-model bingo. Other models do exist. Don't say copilot.

Right. If I can add another anecdote, one of the most fun things about working at Anthropic is that there's just this continuous progression of goosebumps moments where you're like, "Oh my gosh, I can't believe computers can do this." And also, "I can't believe I'm getting to see this up close in the lab as it's materializing."

One of those moments that I'll never forget was the first time that I saw that we had a model that was able to basically start with natural language and create streaming UX, just build charts at the speed of thought, build very capable React single-page apps in a single prompt. I thought, "Oh my gosh, this is going to completely transform how we deliver user experiences." This has progressed into this idea that there's a UI paradigm that you can actually build.

One of the longest open issues on the Grafana repo was, how do I start with natural language and end with streaming charting and BI and structured data presentation? I feel ridiculous saying this out loud on a Grafana podcast, but it just does seem like the wind is at the back of that particular model, that UX design model right now.

It's incredibly painful, batch-editing things. Because from a UX perspective, actually doing good batch-editing UX is really hard, it turns out. I now just believe natural language is the best UX for doing certain tasks, like batch-editing dashboards.

Yes, absolutely.

In one of the early demos that Mat gave me for the Assistant, which I thought was very impressive because I understand how hard it was to achieve, was to make all the themes for the panels purple and change all the panels. Obviously, you show that to a salesperson and they're like, "Why do I want that?" But for an engineer, it's like, oh wow, that's really difficult to pull off.

And suddenly, there's a lot of pre-AI dashboard slop around, but now the AI is actually better at producing dashboards, we've found, than humans. It labels the axes. It puts reasonable names on them and titles.

Right. It yearns to be very thorough. One of the fun things about seeing the models in Claude Code evolve over time is that they've developed this very strong drive to reach what a senior engineer would call completion. You've got it documented, you have test coverage, you have internal calling-convention consistency, you don't have a bunch of dead code in there, you've scrubbed the bad file or the test files.

There are some goosebumps moments where a very narrow set of things it can do has reached superintelligence, like superhuman capability. Security research is a good example of that. Building charts is a good example. The very first example we found is that Claude writes better MCP servers than humans do. No surprise, because it's highly trained on it and it's a foundational technology.

Yeah. We've just launched a new project called GCX, which is our kind of MCP CLI hybrid for all of Grafana Cloud's APIs, and that was written by Claude Code. We basically said, "Please write a CLI MCP server for Grafana Cloud," and here you are. That's how the project got started. It needed a little bit of help. It struggled with a few of the refactorings. There were a few late-night coding sessions while I was in a hotel in Melbourne. I tend to get most of my coding done now when I'm terribly jet-lagged, and honestly Claude Code has massively helped there.

When are we going to get local models? When am I going to be able to Claude Code without Starlink on a plane?

Well, there are certainly harnesses and open weights models that would let you simulate that experience.

Yeah, I get not asking Anthropic necessarily. I just think the rate at which agentic coding is progressing is faster than the rate at which commercial airlines are getting started. Can I get an M5 Max with Qwen and OpenCode, and is that a reasonable experience now?

Yeah, it's surprisingly good. I think it lacks that long-run intuition of the strongest coding models. That's kind of what I've been seeing on the web. There's a feeling of using a small model where you're like, "Oh, you tried." Conversely, there's this experience of using a large model where you're like, "Oh my gosh, I can't believe you're keeping track of all this. I can't believe you managed to do this huge refactor and you one-shotted it."

I think there's a hardware component to it too, which is people have different levels of tolerance for bringing a device onto a plane with 512 megs of VRAM and enough horsepower to make it a good experience. Sure, you could do it. There are small coding models that you could probably run on very high-end portable hardware. That's a lot of work to solve a very localized problem. But when you're on the plane and you want to have the joy of coding, maybe nothing else will do.

One of the fundamental limits will probably be the power supply in your seat is only 100 watts. You'd need a battery backup that you charge in between runs.

I've had that happen, actually. If it pulls too much current, it'll shut you off.

I do look forward to the time when the captain turns on the "please turn off your models" sign.

Yes. And there's a little hack. There's enough capacitance in a lot of these bricks that if you plug it in, let it trigger, and then unplug and plug it back in, eventually you can ride the curve up until you can sustain the current.

Oh, right. Interesting. I've got a 140-watt multi-charger, and it just trips every airplane I've ever been on.

That might be a little ambitious.

Yeah. Please stop running large language models.

Yeah. The engines can't keep up with the electrical drain.

You in the fifth row with the external GPU, I'm talking to you. But when are external GPUs going to have LiPo batteries in them? That's what I'm asking.

That's far from our domain, but they're getting shockingly good, shockingly fast, along with pretty much everything else touched by the AI revolution in some way.

Yeah. So if someone out there is listening and making an external GPU with a battery so I can code on a plane, my email is tom@grafana.com. I'm going to get a lot of spam now.

So, we talk... We obviously have all witnessed the revolution in software engineering. I don't think, at Grafana Labs, any of our engineers are writing code anymore. Our penetration for things like Cursor, Claude Code, Codex, and so on is basically now at 100% in our engineering teams. But I guess this is an observability podcast and we should probably talk about observability at some point. As you go and talk to customers and execs and the community, what are you seeing in terms of the agentic use cases in observability?

It's a very complex set of actors that are gradually reaching consensus. If I go back to what's happening with coding agents, I've seen this real shift at the executive discussion level over the last six months especially. Six months ago it was, are the models going to get good enough? Are the coding agents actually going to get there? Is this something that we should inflict on our team, or can we kind of hang out in shadow IT land and some people come in and use it?

Now it has definitively shifted to, we're creating so much output that now coming up with governance models... If you equip all of your frontline engineers with a fire hose of output, all of your other systems are going to crumble under the amount of stress that they're putting on them. Now it's moving to a second-order problem of building heavy automated test coverage, which of course coding agents are great for, obviously having evals around anything that is non-deterministic, and basically stacking the way up to being able to deploy coding agents with some confidence.

Then the second realization, as orgs sort of get the baseline of internalizing officially blessed coding agents, whatever flavor they pick, is this realization that integration just got really, really cheap. Being able to take one internal system and string it to another one became a prompt and a one-shot and some smoke testing, as opposed to, if you're outsourcing it, massive spec document, throw it over the wall, several iterations, months later, you get this dashboard that has been scoped down to the shadow of what you were hoping it would be, and you're like, "I guess I'll do another turn."

The ability to wire all this stuff up just based on immediate need, and then just sort of chat with your data or build a dashboard based on some wild idea, I think this is putting even more pressure on strong, opinionated UI systems for delivering this stuff, and critically, ways to persist it. One of the uncomfortable flip sides of all this prolific output from product managers and people who didn't think of themselves as coders, but they can write the requirements and they can vibe-code their way to stuff, is distribution and hosting is actually really hard.

If you've got these deeply rooted systems of record where all of your tribal knowledge and all of your state info and your telemetry lives, and you've got kind of a consistent way, for example Grafana, to deliver this to users, the integration is where the magic is happening. It is not just ripping the whole thing off and saying, "Well, anybody can have a database, and we can vibe-code all of the integration layers in between, so why would we need a rendering system?" People still need a way to onboard onto using a platform. It's like back to that old "Who Moved My Cheese?" If the UI is continuously shifting, you can't enable people on it. You can't train people, and you can't really document it.

There are these endpoints of systems of record that I think remain entrenched and are critical business systems. And there are documentable, maintainable, high-uptime output systems, like delivery systems for the insights and the charts. Once an organization internalizes coding agents, builds enough governance to be able to manage them, and kind of unleashes them in a thoughtful way, now you're connecting everything to everything. That's where I think the business breakthroughs and the actual business value starts getting created.

Yeah, no, I understand. I agree. One of the things you said there that really resonated with me is the shadow IT comment. We never really had a problem convincing engineers to use these tools, to be brutally honest. There were maybe a few laggards, but I think what it is to be a software engineer is to naturally want to learn a new set of tools. It used to be every few years, now it's every few months or weeks, but finding ways to be more effective, to be more productive, to get more done more quickly is the engineering mindset. It's the software engineering mindset.

But I find it interesting in the SRE and DevOps space, because I think there's a bit of a mentality shift there. Inherently, to a certain extent, SREs are paid to be and held accountable to be conservative. They're responsible for uptime, availability, performance, cost, and all of these things. They're almost inherently a little bit more afraid of change, a little bit more conservative. At least for me, that's my own internal reasoning as to why the adoption of agentic tools for operations has been a little bit more laggy, a little bit slower. Is that something you've seen as well?

Since it's a podcast, I'll digress a bit. I want to go back to a previous point you made. At least in highly regulated industries, very large companies where software is kind of the back of the house, not digitally native businesses, there is a very high degree of arms folded, lean back:

"Well, wake me up when it can do the difficult thing that I do in this esoteric code base."

This idea of a stampede to adopt these things, I think that's definitely a property of the early adopters by definition. But I think those of us hanging out at the frontier can really understate the size of the area under the crossing-the-chasm curve. It's possible this technology category hasn't crossed the chasm and that we're not even in the early majority yet. There's a degree of socialization where you have almost another normal distribution. Some people adopt it and really dive in with both feet. Some people are at the other extreme end, very skeptical.

I think you could probably put operations on the back half of that, because there's not a huge amount of glory and credit in adopting something new, but there's tremendous downside in taking an unjustified risk and then causing an incident of some kind. I think that's going to be one where I've used this analogy a few times with different aspects of this: it's almost like flight hours to build a degree of trust. These are non-deterministic systems. They operate at different levels of effectiveness in different environments, and they're not assertable.

You can build an eval, but you can't just say, "Yes, this is symbolic code. It always works this way every single time. You can expect it to work this way in production." Once we've gone through various tests, you can kind of build on this as a foundational piece. AI is very fluid. You might have regressions from model to model, although hopefully with Claude models you don't, but you're kind of targeting this almost human performance bar. The only way to persuade some people is just to have lots and lots of successes and build them up from a low base.

One of the habits of rolling out these tools across diverse populations of engineers that I think has worked pretty well is it's very tempting as an engineer to look to glamorize the hardest problems. I always get this secret thrill when I'm using Claude Code and I discover that it hit its complexity limit, and now whatever it is I do still has got to take over and coach it and steer it in the right direction.

There are all these things that are not the fun part, and they pay the bills, but it's hard to retain engineers to do. It's hard to motivate people to stay focused on it. It's hard to maintain quality consistency. This is all the negative space around the cool part that I think is a way to build the trust relationship, to get those successful flight hours.

Things like dragging some crufty old component from Python 2 to Python 3, or doing a framework migration, you're just modernizing a big code base that is very, very testable, and you can just know for sure that you succeeded. All of these things that engineers kind of hate and invisibly route around and build these, "This is a black box now, don't go in there, nothing good's in there," it kind of warps decision-making, and I think this affects almost every kind of team.

Finding things that no one will miss, and finding things where there's just a bona fide win, that this thing that everybody hates got a little bit better, that's a way to build towards having the most extreme maturity level, which is probably what we're in at Anthropic, where almost every stage along the way is Claude-powered or Claude-automated in some way.

That's the only way that humans can scale across all the stuff that we're trying to do. I'd say we're out there on the frontier making mistakes, and other companies can kind of learn from what goes well and what doesn't go as well, and they can adapt that to their own risk profiles and their own engineering cultures.

Yeah, no, I couldn't agree more. One of the things you said was the fire hose comment about producing software. We're going to have to bring agents and automation into DevOps, SRE, and operations in general. Otherwise, how the hell are we going to keep up with the amount of software being written? It's just not going to be feasible. We have to figure out a way that the majority of the toil of updating versions of third-party packages, rolling out changes, rolling back stuff when you break it, and all of this just needs to be handled automatically for the majority of common cases. That's the model I want to get to.

Yeah. On that note, I think one of the most exciting things going on in these systems as they become more complex is really thoughtfully implementing these adversarial relationships between two parts of the system, where one AI's purpose is to make another AI's work better in an iterative cycle. Having systems that are wired to work against each other in the spirit of a common goal is actually something that human organizations do really well at scale. It's a great pattern that I think is a glimpse of how we can get this under control in some way before it all changes again.

Yeah, definitely. We've seen examples of that working as we build out our Assistant, and we have our investigations piece, which is essentially where we swarm many assistants over a problem and they work concurrently. We have a devil's advocate in there, which is just challenging: is this right? It's amazing how similar they end up behaving to human organizations. The thing is the same, then the same things apply. All the learning we have for human interactions and stuff, think about that transposed into agents and see what you can do, because it is shockingly similar.

That's exactly it. One of my go-to slides in some of the presentations I use is, there's a great quote from one of our founders: the time to use an agent is when you would hire a smart human and train them to do something. You need a cognitive black box to drop into some system.

That's kind of the twin quote to another one that I got early in my career, which I feel like is very relevant these days. Somebody half-jokingly in an IRC channel said, "If you're trying to solve a problem using a computer, figure out how you would solve it without a computer and then have a computer do that." Basically, if you're trying to build systems to organize agents, look at how you would build a human org chart, and then that's a really good guide for how you should build that system, and how you should filter work down, delegate down, and synthesize upwards. Critically, like human systems, they tend to work better with a single orchestrator as opposed to multiple peer agents trying to work it out without an authority relationship with each other.

Yeah, we've also seen that. We're doing different tricks, like we do send them off on the same mission and then look at the end. We see who arrived at the same conclusions, things like that, where we use the non-determinism as a feature instead of something that gets

You know, as Mat has alluded to, we built tons of different agents inside Grafana now, and we're definitely having huge value delivered to our users from these agents. We're blown away. We've gone from being, honestly, 18 months ago, relatively skeptical about this whole thing to complete maximalists. Everything's going to be done by an agent in software engineering, operations, and observability.

It's been a very quick transition for us as a company. We were those laggards, those skeptics that I was talking about. That was us 18 months ago. I've always been curious as to why the transition for us was so quick. And by us, I mean Grafana Labs. I've seen it happen much more slowly at other organizations, particularly other observability vendors. They started before we did and took a lot longer to get to where we are now.

We've always wondered why. What is the reason this technology works so well with our stack and just seems to get it and do the right thing? We don't build our own models. We just do some prompting, build a few agents, put a nice UI around it, and integrate it with everything else.

A bit more than that, mate. Yeah.

Mat's team is actually the one that does all the incredibly hard work, but they seem to make it look easy.

We've come up with this idea, and it's very hard to prove wrong or not, so that's why it's a lovely idea: that your foundation models, Eric, from Anthropic and from other vendors, just know how to use our software. They know how to use our software because we are so open, because the vast majority, 80%, of our software is open source. We've got a 35-million-user-strong community who are constantly publishing blog posts, and the amount of open source software instrumented with OpenTelemetry and Prometheus, and the number of example dashboards out there sitting on GitHub, means your software...

...just seems to know the right thing to do better than any one human does. Internally and even externally now, we talk about how open source, which has been core to Grafana Labs for 12 years now, is the cheat code for AI. Obviously for us this is very important, because we love open source and it's such a reason why so many people choose to work at Grafana Labs. Do you see that elsewhere? Is that true? Am I just blowing smoke? What's your take on that? Be nice.

Don't break his heart, Eric.

I'm not down on the research floor, so I couldn't give a definitive answer, but subjectively, I absolutely see the same thing. These are very complex models. Dario once used a word to describe two unrelated things interacting with each other in some way, and I just love this: spooky entanglement. Everything is connected to everything in the models.

I think of it in terms of domains that are illuminated to the model and domains that are opaque to it. All things equal, having the opportunity to do heavy training on the entire source code for, say, the Linux kernel or Grafana tends to set it up to be more effective at working with that kind of complex system.

At the same time, observing and listening over the last year, I feel like this has been the year of the rise of reinforcement learning and the end of the stochastic parrot analogy of, "Well, it can only regurgitate what it's trained on." If that were ever true, that ship has long since sailed.

All of the RL that we do to build coding capabilities is driving this synthesis capability. So it's two sides of the same coin. Yes, definitely being in the training corpus is a huge deal. All things equal, I found that Claude Code is more effective working on Linux and Unix environments than on Windows. Not by a ton, but a little bit.

But all of the investment in RL is what drives that intuition of, "I understand this pattern. Here's what I should do next." So I think it's sort of all of the above.

Yeah. It's obviously acute to us in observability, but you mentioned the Linux kernel. Are there other places where you're seeing openness being key to leveraging AI?

I think, not to be too close to home, but MCP is my favorite example of this. In full disclosure, early in my Anthropic tenure I was the product manager for this crazy, early-stage, wild ideas team, and I helped launch MCP. We had to figure out what the narrative was going to be, so I got a chance to be in the room when all the smart folks that built it were thinking about the future and chatting with each other.

It is one of the very first feedback loops that I think we entered as a company, where first we solved the technology problem: we need to get context into large language models in a reliable, standards-based way. Then, having thought through that, we arrived at, "Obviously we have to open source this thing. We have to make it an open standard, because that's the way that you get scale. That's where you get adoption." You take all the risk out of some kind of vendor nefariousness by just saying, "Hey, here, it's for everybody."

Immediately after that, it became part of training Claude to use MCP servers effectively. So we solved a problem, created an open source code base, and then began bending the model towards working effectively with that.

I may be mangling this quote, but I think somebody said to me that if you had an API shaped exactly like MCP, with different names in a different language...

Yeah, we see that a lot with our MCP servers, with our tools that we've built.

Funny. If it was functionally the same thing but just slightly tweaked, it would way underperform actual MCP, because there's so much investment in training the models to work with that now open source code base.

That's a good one. The model will guess the name of the parameters, get it wrong, ask what the right one is, and fix it. So if you name your parameters what the model guessed, what's the term used for this? Hallucination squatting?

Yeah. Hallucination squatting.

We kind of did this with Assistant. When we wired it up, we'd see what it guesses and then rename the parameters to be the ones it guesses. Then it's quicker, it uses less context, and it gets it.

The first thing we did was wire up basically Claude to Grafana and let it just chat. It could do a couple of things: query some metrics and look for logs in these very early days. Then it started saying, "I would look for traces here, but I don't have any tools for traces. Otherwise that's where I might go next. Maybe there's a dashboard. It'd be great if I could search dashboards, but I can't. You can." It would tell the user how to do it.

Then we realized that is just our little roadmap. Keep using it, find the places where it tries to do the thing and can't, and then make it so it can do it. It was a really nice experience doing that, and it happened very quickly, which led us to suspect that it sort of knew what we were doing. It intuitively already knew what was going on.

So, yeah, we started to suspect it knew us better than we knew ourselves.

Are we allowed to start attributing knowledge and understanding to these models? Or is that still something I'm a little bit uncomfortable with? How do we feel about that?

I think that's eyes of the beholder, and knowledge and understanding are very different.

Yes, of course. Yeah.

They can encode a lot of knowledge, and it's real knowledge encoded in there. Understanding? Potato, potato. As a colleague would say, one man's floor is another man's ceiling. It's really in the eyes of the beholder whether understanding is occurring, but it sure looks like it when you give it a whole lot of complex data. At any level, anything from "read this chart" to "inspect this binary dump," it happily gets on with it.

Mat, I wanted to come back around to one thing you just said. I think you're describing a design pattern in this era where you can build things very quickly and offer these moments of delight, but what does it add up to? One of the design patterns is: if a user ever has to cut and paste, that's a bug. Really simple. If you are manually carrying context from one part of your system to another, attack that immediately because you're inserting a human operation into an automated loop.

A lot of the way to think about these things is wherever you are slowing down the ability of a loop to iterate towards a solution, you should automate that thing immediately and let the loop do its thing. That assumes that you have something checking the work that then feeds into that iteration. It's simple: you just use ever more AI in an escalating ratchet.

This is actually a really good segue to my next topic as well, which is around security and AI, specifically in observability and operations. If you go and talk to any of our customers adopting the Assistant, the first thing they say is, "Okay, the Assistant will tell me what's gone wrong with my system and will tell me how to fix it. Why can't I just click and have it go and fix it for me?" Then the next thing they'll say is, "Oh, but I want to make sure there's a human in the loop and checking that."

They want to make sure someone is there. You can't hold these models accountable, right? If they make some mistake, make a change, cause an outage, there's no... I can't sue Anthropic. Maybe you're going to change your terms and conditions to allow me to, but I doubt it.

Just paste the contract or the lawsuit in and see what it says.

It'll win.

Yeah. This is my fault.

When it comes to checking models, it strikes me as rife with issues, because one model will trick the checking model with various attacks. You already see this in the coding agents when you say, "Make sure all the tests pass," and if one of the tests doesn't pass, half the time the agent just comments out the failing test instead of fixing the code.

What's your approach to introducing safety when there are real world consequences here? I'm not going to go to the trolley cart problem, but just the, "Oh, I could cause an outage in my little application."

Well, it's defense in depth. At a certain point, your models should not be biased towards writing a fake test implementation that doesn't actually satisfy the spirit of the test. That's a model alignment-level thing. It just should not do that. You should be using high-quality coding models.

These are tools, not a panacea. Again, that engineering temptation to reach right for the grand prize as opposed to build up brick by brick, I think, is the shared enemy here: going too far too fast and delegating too much control.

Yes, yeah.

The fundamental thing that I see reinforced across all these different conversations is, at least at the executive level, at the governance level, we are firmly wedded to the idea, for now, of human accountability at the root. If you as an accountable leader stand up a bunch of agents and let them rip, and they do something that perhaps eluded a human review step or was classified as not requiring human review and it made it out to prod, guess who's holding the bag?

It's you, the accountable leader. You have deployed some tools. You've created this business outcome. For better or worse, you're accountable for it. I think the idea of delegating to an agent the responsibilities and accountabilities of a human employee, if it ever comes, is way, way out.

Yeah, no, for sure. That is almost a cultural shift before it's a technology shift.

That's not even entering the conversation. Folks are interested in the abstract philosophy of it, but nobody is considering suspending human accountability systems. For better or worse, that's how we organize ourselves.

Right. You mentioned a lot earlier on about highly regulated industries. Even we have some regulation, SOC 2 and ISO whatever, that all our code changes have to be reviewed by a human at Grafana Labs. We don't let agents review agents and then auto-submit. We put in place checks and balances to make sure at least one human has read this code and approved it, and they are therefore accountable. You have some pretty strong rules in the Assistant team, Mat.

Yeah. It ends up being two people, because the person that does the PR with the agent has to pay attention to that, and then another person is reviewing. We do have that. There are times where I have a command to just babysit CI. It'll build up some nice new feature that I got together very quickly, and then up it goes.

Of course I do linting and things locally, but inevitably there are extra checks that are happening in our CI. There are actually a lot of checks because we inherit all of Grafana's checks as well. It's a big open source project that's grown and has quite a few checks and balances in place in its own right. So I have an agent just watching that, looking for the comments, having a look to see if there are tweaks, and it'll make little changes that it then commits.

Right. So you have this hard constraint of human review of developer intent.

But at the end of that, before I say it's ready for review, I then review everything. Most of the time nowadays, those little tweaks it makes are exactly what you do in response to that specific error message. That is what we're doing.

Yeah. You're bound by the constraint. Everything else is the negative space around that constraint, and you're filling it in with things that make you faster and more effective, which I think is the pattern.

Okay. I would love to look at the concept of operational toil. I get woken up in the middle of the night by... We don't actually put engineers on call at night, but hypothetically, one gets woken up in the middle of the night.

They might be having a sleep in the day.

That has been known to happen with the amount of plane rides I take. The jet lag's pretty bad. I have a couch in my office for that reason.

I get woken up in the middle of the night, and the Japanese cluster is paging me because queries are failing. The Assistant has already told me it's your gRPC buffers, or whatever, that need resizing. It's even told me exactly what I need to change. There's a world where the next step is an agent goes and makes that change in our production repos for us, and I just have to read it and click yes.

If you look in that repo with all of the config for everything in Grafana Cloud and for every region, we already have checks for blast radius. You can't make a PR that touches multiple services or multiple regions. We already have checks for cost. Every PR gets an estimate of how much it's going to cost to deploy, and there's a check. You can't go and spend a million bucks. The agent could never scale a job up to a million.

We already have all of the linting, and these are all deterministic. None of these are agent-based checks. I can imagine a world, as you said with your flight analogy, and I love the flight analogy by the way, because the CEO of Grafana Labs is a pilot, and half of the stuff internally in Grafana Labs has flight analogies to go with it.

I could see where you've built up enough confidence that we've just rubber-stamped so many of these tiny little production PRs that we actually just say, "Look, the checks we have in place are enough, as long as the PR is no more than a handful of lines and passes all of our checks." None of these checks are agentic. There's no opportunity. I mean, I guess it could comment out all the checks. Maybe we should have a check against that somehow.

There's very little opportunity for malicious activity here. I would love to see a world where I don't get woken up, unless the change it proposes doesn't pass the checks, was too big, was too risky, or it can't figure it out. I'd love to see that de-amplification of trivial...

The blank space around the hard problem, right? You're building your way there. You have incredibly sophisticated and comprehensive deterministic CI test coverage. That is an incredible foundation. That's the first thing that everybody has to build in order to admit these things into the pipeline in some way.

The problem you're framing is: how many wins would it take me to trust it with the next tier of risk? If you have this foundation of being able to de-risk things programmatically... Actually, this reminds me of a story that one of our engineers told me way back. In the early days of Claude Code, when it was really starting to get good, our production inference code base was weakly typed Python. He woke up one day and said, "This shall not stand for one more day."

He gave Claude the task and said, "Implement strong typing across this entire code base." Here's the critical piece: "Build 100% test coverage and then build tests to test the tests." Essentially, he dumped tokens into this problem until it was taken to an almost ludicrous extreme of de-risking.

That was the foundation for the team to be able to say, "Okay, now we're overlaying the human checks and the human senior engineering and architecture quality process on top of this thing that has already been extensively de-risked by knocking out the easy stuff."

The way the story ends is within the course of a week, they were able to get into prod with the now strongly typed code base, which is a huge regression opportunity. It worked great. Then more people became true believers, and more of us kept writing code with agents.

So that makes me think, because we have the same thing with investigating root causes and things. We can burn some tokens on that. Maybe if it's a low-risk item, you don't want to burn as many, but for those ones that are really serious, touching your SLOs, or customers reporting outages and things, maybe it's worth it to really burn some tokens on that problem and dig into it.

You have tiers of classifiers.

Yeah, I was going to say, you have these tiers of classifiers.

Sorry, I didn't mean to interrupt. Exactly. The easiest layer is you just add some classifiers, like hard problem, easy problem. Then at a certain point you get to the question: should we just dump tokens into this problem to try to optimistically find a solution, or do we page somebody?

Right. I'll tell you what, though: it is very exciting technology to work with. It does what Tom mentioned earlier, that engineer spirit. It's like we're starting again, and we have to learn a completely new technology that just behaves differently and is therefore capable of doing amazing things.

The thing that sold it to us was using it in really targeted ways. We were very specific. I saw some people come in with the big high-level thing, give it some really difficult task, and inevitably it couldn't do it in these early models. Now it probably could do a lot of them as well.

Starting small as a user, just use it to assist what you're already doing. Now, if I'm engineering something, I do tend to have a few agents on the go at the same time. I do a lot of planning. If you do a lot of detailed planning, you can let the agents run and crunch on it if they have that feedback loop. You just have to describe this stuff to it and make sure it has the things it needs if it's running local commands or whatever it's doing to run your tests.

It turns out that works really well, and I'm excited to see this transposed to other places, but I think it's the same principles.

My favorite use of this... I don't get to code that much. I've got a very large engineering team. I'm mostly a manager now, and I have a backlog of coding projects that I've wanted to do. I'm slowly ticking off that backlog.

This is the intersection of AI and observability, because I do a lot of home automation. Everything in my house is fully automated, and I really want to store a lot more telemetry about my Zigbee mesh, for instance. Sometimes you press a button and it doesn't quite work, or doesn't work as quickly as you would hope, and I want to know why. What do I need to optimize and fix?

I just opened this PR yesterday, actually, and it was mostly Claude Code. It's been on my backlog to go and instrument Zigbee2MQTT for a very long time. It's not hard. It's 500 lines of Node. I am not a JavaScript or TypeScript engineer. It would have taken me days, if not weeks, to have done that, because I would have had to learn a ton. But I can read it and it seems reasonable. It does the right thing. It's got 100% test coverage. The satisfaction of being able to tick off a bunch of personal projects, with tokens very graciously donated by Grafana Labs, has just been absolutely huge.

As it turns out, this podcast mostly turns into Mat and I talking to each other, but no, please.

Oh, sorry, Mat, go ahead.

No, please reply to that.

No, it's great, super fun. There are two really interesting principles, not principles, attributes of coding agents for certain personalities. One of them is, I've also gone down the rabbit hole of Home Assistant and linking everything to everything and automating it all. I use Grafana and my Home Assistant to look at my InfluxDB, which is all my sensor outputs and so on.

Terrific. A hot tip: if you use the Prometheus exporter in Home Assistant, I recently refactored all of that, again using Claude Code, two months ago, so it's now got loads more entities in Prometheus.

Excellent. I know, I'm turning into a serious Home Assistant nerd, so perhaps there's a separate conversation. The thing that is sort of the hallmark of my journey through it is, it's a hundred-year-old house with layers and layers of obsolete tech that I gradually integrated into the system. There were certain problems along the way that were absolutely not worth, even as a hobby, even as a random couple hours on a weekend side project. It would simply not be worth the time to wade into it with the current technology and my current understanding.

This zone of things that I can trust Claude to solve within is now a fairly complex software project, but the only upside, there's no business value generated. I can click something on my phone that I used to click on a wall panel, right? Things like that. Or I can see high-fidelity data that I couldn't see before. The cost benefit is getting completely transformed in terms of the effort that I imagine something is going to take. It has been in steep log decay to the point where basically nothing feels out of reach in this sprawling home integration project. It's because I've built this confidence that between... I've got CLAUDE.md files everywhere. I'm using the same sort of encapsulation and factoring approaches that we would use in an enterprise code base for no good reason, just because it feels good. But there's nothing in this code base that I think is not in the realm of, "I would give Claude a crack at it." That was not the case a year ago.

There were certain things where I was like, "Okay, well, I'm just going to grit my teeth and reach in there and write the code, or I'm just not going to do it at all. I'm just going to wait." So the first property is, many things that were total wastes of time got so cheap that just dashing off a Claude prompt and then checking in an hour later and saying, "Oh my goodness, I got this thing. I now have this technology asset that I didn't have before." I wasn't expecting that pure upside.

The second one I think is really profound for managers, and I experienced this. The last time I wrote production code was five years ago. It was the whole experience of being in the flow. You're in your dev loop every day. You're always synced up. You're ready to rock. You sit down, your dev machine is ready to go. You feel confident in your estimates. You're burning down your backlog.

There's a half-life of the quality of your dev situation where I can step away for a month and come back, and it would be eight hours before I was back in the flow. You guys know how it goes. You pull down the latest, you've got some break, somebody forgot to check in this other thing, there was a framework migration, now you have to go read the docs for this framework, just on and on. You've probably experienced this: the longer you're in that management zone, the worse that tax gets until one day you're like, "You know what? It's not worth it for that one moment of delight where I wrote a three-line check-in." It just fails cost benefit.

The ability to ramp back up into the flow is almost always instantaneous now, because Claude will just bash through whatever nonsense is keeping you from being in your dev loop. So I feel like on one hand, it's this incredible opportunity for builders to pursue kind of marginally valuable things that might yield something really high value, but are kind of risky or expensive. For people at every level of an engineering org and every level of a PM org, it's a shorter path to those moments of delight that get us all into coding.

We refer to managers in Grafana Labs as recovering engineers.

Yeah, right.

It's the encapsulation of exactly that idea. Suddenly that recovering engineer can get the dopamine hit of solving a problem that they didn't used to be able to.

It's incredible.

It's not just managers, to be honest. I have never contributed to Zigbee2MQTT or Home Assistant until now, because I don't know Python, I don't know Node.js, and it would have taken me too long to get a dev environment working. With Claude, I can read these code bases quite easily. I can make sure my contributions are still reasonable. I'm never going to submit slop. It's so satisfying. Now I've got two or three contributions under my belt, and more and more, I just fire off a prompt and it can do it.

It's not just engineering tasks as well. Are we allowed to talk about using Claude to do our performance reviews yet? As managers, we have a repo of a knowledge base of stuff, and we all check out that repo and use Claude to talk to it. So it's not just engineering tasks where you can start to get the help from models.

Move on from that topic.

It's a tool. It's all in how you use it. I don't think I'd want Claude writing my performance review, because when given enough information about how humans spend their day, it's actually a bit ruthless. Careful what you ask. But being able to pull together a whole lot of unstructured data in order to form a perspective, who wouldn't want to start there?

Yeah. We ingest these notes from a meeting, and it can go and update all the necessary files, all the projects, all the people. We have a markdown file for each person of just stuff they're interested in and focused on, so it is just like having a memory.

The nice thing is, because it's a GitHub repo, we get PRs. We can actually pull up a PR and say, "Right, here's the meeting ingestion. What do you think?" Comments, tweaks, and stuff. It's fantastic. It's the next thing, but it's not tracking usage or GitHub stats. It's literally just note keeping for us.

If you think, "I need to talk to Tom about this thing," I'd have to go and find Tom's file and add it to that thing, but it might be four people I want to talk to.

The unexamined life is not worth living, but this is a lot of examination.

Are you telling me my file is not always open in a tab somewhere?

No, yours is a favorite. I was talking of a different Tom. It's a different Tom's that I keep in the trash. Let me ask you this, though: do you think we will end up in a situation where we've stopped looking at the code...

I mean, "we" is doing a lot of hard work in that sentence.

...like assembly? We don't really look at assembly unless we need to. Most people can't, though they probably can now thanks to Claude, etc. Do you think we'll get to the point where the code is like, you look at it...

If you need to debug something, otherwise you're good. You know, if you talk to Boris, the creator of Claude Code, his view is, "We're already there." About a year ago, I stopped dirtying my hands with reading the actual code. Now I operate at the pattern level.

Increasingly, lately, I've had this weird experience with Opus 4.6 where I'll think I see something smart and I'll interrupt it in a loop. Then it dawns on me that it's actually a step ahead of me, and I'm like, "Oh, I'm so sorry. I thought I understood that, but actually, I don't. You're already on the right track."

So there's a certain threshold where, you know, I fancy on a good day I'm a decent engineer and I kind of know what's going on. I had this very uncomfortable feeling of being in Claude's way as it was trying to solve the problem and benevolently deliver the thing that I was asking it for.

That's funny.

One tier of question is, should we go review the code? Another one entirely is, is a human, even a competent one that knows the code base, for some value of competent, adding value or reducing throughput? I think these are the questions that, back to the idea of software engineering collectively engaging with it, there's no one right answer. Again, it's a question of risk tolerance.

But I've definitely had this ratcheting sense of my value being pushed out of implementation in the same way that anybody that's ever written assembly would feel that most likely a high-level language expressing programmer intent very well, and a strong performance and compilation stack, is going to outperform whatever any of us could do at the assembly level.

Yeah, I see that future. I really do. It's closer than we think. Very exciting.

No, I think that was brilliant. Hopefully this bit won't be included.

Incredibly fun session with you guys. I haven't had a chance to do...

Okay, is there anything else we want to talk about before we sign off? We're kind of out of time.

I don't know. That's normally how we finish the podcast.

Again, if...

Yeah, it's just like anything. All right.

I know it can be nice.

Oh, no, wait, wait, wait, Eric, we do have to. We have an actual way we finish the podcast.

Oh, okay.

Be nice. It can be nice. I'd like to have this on record, Eric, you were saying...

No, talking about AI governance in the corporate boardroom is very different from having some free association with a couple of deep tech guys.

No, we appreciate having you, yeah.

Yeah, thanks for joining us. It's good to hang out.

And thanks so much for all the insights. It's fantastic.

It's a great forum to be able to join.

And thank you for MCP. I mean, if we can swing it, Tom, can that be the strapline? "Thank you for MCP." It'll be something that's going to clickbait. Okay, you're listening, Tom.

Well, I can't accept that because I was just in the room when some very smart people put it together. But, you know, I got to say a few words here and there and maybe help it along its journey.

Yeah, it's very fun. I could see that you're having a lot of fun there.

I think it was just a very cool bunch of rooms to be in at the time.

It's similar vibes, I think, over in Grafana Labs. We're cooking with this.

Mm-hmm.

We're really moving fast and having a lot of fun doing it.

Yeah.

And you don't lose... I was worried I was going to miss the day-to-day coding bit, but actually, for me, it was always about building products, and that's just got a lot easier. So yeah, I'm all in on that. Thank you so much, Eric Burns from Anthropic, for joining us, and thanks for listening. We'll see you next time on Grafana's Big Tent.