**Mat Ryer:** \[00:00\] Hello and welcome to Grafana's Big Tent, the podcast all about the people, community, tools and tech around observability. I'm here with my co-host Tom Wilkie. Hello, Tom.

**Tom Wilkie:** Hello, Mat, how you doing?

**Mat Ryer:** Pretty good, thanks. That theme tune's good, isn't it? You like it?

**Tom Wilkie:** I do like that. We were just having a chat about it before and it really gets me in the mood for the podcast.

**Mat Ryer:** Yeah, you need it. You actually refused to do the podcast without it, don't you?

**Tom Wilkie:** I do, yeah. Did you enjoy your little trips to the South of France last week? Was it last week?

**Mat Ryer:** It was last week. It was a lovely, beautiful place. Nice food, nice drinks, lovely people.

**Tom Wilkie:** Oh, I was just going to say I hear about the people, yeah.

**Mat Ryer:** I'm actually going literally to the same place tomorrow. So, yeah, meet me as well, I'm flying to Nice on Friday.

**Tom Wilkie:** Oh, blind me, we'll be there.

**Mat Ryer:** Okay, no, didn't realise that. So bougie.

The voices you just heard, let's introduce them. We're also joined by Torkel and Ryan from Grafana, the Grafana team. Hello, Torkel, how's it going?

**Torkel Ödegaard:** Hey, hello, well, it's going pretty well. Summer is over.

**Mat Ryer:** And did you have your day of summer in Sweden, did you?

**Torkel Ödegaard:** Well, more like a month, but we'll take what we get.

**Mat Ryer:** Yeah, it's just top up your vitamin D and then you can go back in for a year, another year.

Ryan, you also, Ryan, didn't you start contributing through open source to Grafana? Isn't that how you got into Grafana?

**Ryan McKinley:** Yeah, so time flies, but I was working a while back with an experimental hydropower company. And we were looking at various ways to build control systems and monitoring and found Grafana in its early days. And I started working a lot with the plugin ecosystem at the time and built our entire control system and visualization based on Grafana. And then many years ago, we began this Angular to React migration, which changed all the things, which pulled me in deeper to make sure we could continue to build our custom applications within this platform. And I kept working more and more with that. And my contributions in the software side were probably more interesting than what I was doing for the hydropower. So I really enjoyed working with Torkel and the rest of the team here. So joined now six years ago. So wow, off we go.

**Mat Ryer:** I really scares me that Grafana's being used by power stations.

**Tom Wilkie:** It's more than that. Yeah, well, the space ones as well recently, right? Like, hopefully nothing in the critical path though.

**Ryan McKinley:** No, but we do are though, because I was trying to the guy from the Japanese space agency. And they were saying that essentially, they can't see anything except for Grafana. That is where the data shows up. So that is literally their view into what's happening when they put things on the moon and that's just kind of amazing.

**Torkel Ödegaard:** Definitely not something that I imagined when setting up, sort of creating Grafana, is going to be used for power stations or spaceships.

**Mat Ryer:** What was it for?

**Torkel Ödegaard:** \[04:03\] Well, I mean, funnily enough, like in the beginning, it started with application level metrics. Like I really wanted to get insight into a microservices system I was building and simple stuff, like sort of performance and queue, how the sort of messages were being processed and what the queue depths were. But then quickly, actually, we started using it for business level metrics as well to sort of, and the thing that I really fell in love with is being able to correlate, like sort of looking at, like this was an auction site, like sort of being able to see how the amount of user bids and the value and how many people logged in and correlate that with, like, yeah, with their performance or if you changed something on the site, like being able to see the real time, the impact on the business of a code change or a performance change or an outage.

Like I remember in the early days, like I actually caused an outage on this auction site and I could so easily in sort of in Grafana and with the Graphite sort of metrics where I would compare to the week to the sort of, yeah, but I'm not sure what Wednesday or something, the Wednesday or the week before, what the total kind of bid revenue or something was and I could compare, okay, this outage cost us X amount of dollars and I could sort of plot that and in real time or like very sort of quickly, so look at that and what the cost of the outage was because it was so easy to sort of go back a week and compare exactly that sort of the time frame of the outage and just being able to do that with, and put that on one side kind of the normal application metrics was really to me super exciting.

**Mat Ryer:** Yeah, you realize putting all those things together in that one place, yeah, unlocks quite a lot of stuff. Did that really help sell it then? Would that really like help convince people that this was useful when you have these outages and you suddenly you've got all these insights that they didn't have?

**Torkel Ödegaard:** \[06:05\] Yeah, I think, funnily enough I think that the thing that sold it the most was actually sort of ease of use and looking cool. Compared to Nagios. Yeah, and compared to most other kind of sort of observability metrics tools like that had small image-based sort of image-based graphs that you couldn't really interact with, but really time consuming to set up and very limiting. Just being able to sort of have these colorful dashboards that were relatively easy for anyone to create and looking good enough so you really wanted to have them on a TV, like that's the thing that sort of really made it so that everyone, every team in the company walking past or seeing sort of other teams' dashboards, but oh, I want this too, we want this as well because it just looked cool and was relatively easy for them to do that.

**Mat Ryer:** Do you think the move to remote working has kind of impacted Grafana adoption because of that?

**Torkel Ödegaard:** No, I actually think so because like, I think the thing that really spread Grafana, I think, in the early days, was seeing other teams, sort of, TVs, like walking past them or, so I think, in some sense, I think that the work from home has reduced that sort of visibility into other teams, kind of, what they're working on and seeing their TVs mounted, kind of, on the wall as you walk past them.

**Mat Ryer:** Yeah, so was that intentional when you built this? You didn't need to make it look good, right? You could have, it could have just been black lines or something on one page.

**Tom Wilkie:** You clearly don't know Torkel.

**Torkel Ödegaard:** \[08:06\] Well, it was intentional in the sense that I, this was maybe the third kind of UI project that I built at that point. And I just, I, for me personally, I like working with software that looks good. And so that's just a preference, like I know that I'm going to invest more time in it. I'm going to enjoy using it more. If it looks cool and looks like, well, designed. So that's what's mostly working towards my own preference, like if I, if I won't be working on it and using it, maybe I'll just, yeah.

**Mat Ryer:** I agree with that, I think people still underestimate that. I think the whole web 2.0 thing showed us that websites actually can, like, look pleasant as well and they can be usable and easy to navigate and all this. And you started to, that's when I really started to pay attention to that front end. And the fact that you can, you can have two bits of software that are identical. But the one that has the more beautiful interface, I feel like it's just going to win. Like it has, yeah, we're still humans and it still matters. So yeah, I agree with it. And I always appreciate it. I saw some pictures of early, early Grafana and it did from the beginning, it looked great. And I think, you know, why not?

So this is, this is now, because we at this, I'm, this podcast is about the 12th release of Grafana, right? Can you believe it's been, it's been 12 releases now. Like did, did you ever think we would, you know, still be working on this project 12 years on, Torkel?

**Torkel Ödegaard:** \[10:08\] No, far from it, like, I mean, before I started Grafana, I think the longest I've been working at a single company or project or product was maybe two and a half years or something. Yeah, so like, yeah, starting out something that I would be working on, yeah, 12 years later, yeah, was very far from, from my mind.

But one of the things that I enjoy about that and I think that sort of carries over from my time before Grafana is that I was, I cared a lot about sort of the main code maintainability and like thinking about architecture and systems like that evolved and change. And yeah, so like I used to be very, so highly into sort of extreme programming and test driven development and thinking about code that is easy to maintain and change. So having worked on a single code base that's evolved for 12 years has been very rewarding and being able to learn from that past kind of Grafana life and try to apply it to open source, which is also a very different kind of environment has been very sort of interesting and rewarding.

**Mat Ryer:** And also like, you know, must be rewarding working on a piece of software that's reached so many people, right? Like I definitely don't think I've ever worked on a piece of software that's been used so widely in the industry. Like how many, what is the current kind of, at least publicly released number for the number of Grafana users?

**Torkel Ödegaard:** Yeah, I think the latest number is things like anywhere from sort of 25 to 50 million, depending on how we count sort of, and depending on how we count sort of active users and not active users, but it's, I think the sense is, it's a lot. It's like a lot.

**Mat Ryer:** It is. Software engineers are there in the world, right? You know, we might have a fair proportion of the market, right?

**Ryan McKinley:** It's also all the people trying to understand what the software teams are doing, not just the developers themselves, so.

**Mat Ryer:** Yeah, I think that's a pretty, yeah.

**Tom Wilkie:** \[12:08\] That's a really good point, Ryan, like it's not just developers.

**Ryan McKinley:** It's a pretty big sort of set of potential users when you include sort of product managers and every person in that company that sort of cares about. There's so few projects where the whole team, the whole company doesn't care what you're doing. So different levels of what parts are traditionally observability versus business metrics, who knows, but the audience is huge.

**Tom Wilkie:** Yeah, no, for sure. And I guess if you could put your finger on, like one thing in Grafana that's led to this, like, level of usage and the growth and like the engagement that we have for the community, what would it be? Like, leading question, I kind of know what I think it is.

**Ryan McKinley:** For me, it's that it looks good and allows you to extend to whatever your systems happen to be. And we all know those change over time. So that's part of our pluggable open framework where this started off driven entirely by Graphite. Graphite was super important, but over time, Prometheus became kind of more important. And we're seeing new data sources pop up, new ways of visualizing stuff, who puts their data in which sources. Like currently, we support about 200 or so different sources of data, which kind of allows teams to work with their data wherever it is.

**Tom Wilkie:** Did I see correctly that in aggregate, all of the SQL data sources are now more popular than the Prometheus data source? Like in aggregate, are they just growing more quickly in usage or?

**Ryan McKinley:** I haven't looked at the recent numbers, but that sounds about right. And the flavors of SQL are amazing. I don't know if you'll see some of the most interesting AI projects now tend to use SQL as their lingua franca, because we all know I don't want to learn to write LogQL, but you can have AI write so much stuff based on SQL style languages.

**Tom Wilkie:** \[14:08\] Yeah, I think it also speaks to the growth in Grafana usage being outside of traditional IT observability and software observability, I think. There's not a lot of monitoring systems using SQL as their query language.

**Ryan McKinley:** Yeah, and they evolve and change all the time. So part of what's been great about Grafana is it lets the teams evolve as the backing technologies improve and over time.

**Tom Wilkie:** Yeah, and I think also the usability improvements over time as well must be opening up to more and more people.

**Torkel Ödegaard:** Yeah, because, you know, with Explore, I mean, it started, it was kind of a pro tool in those early days, and there were a lot of people, so many features, it's all packed in, so much power and so many things you can do, but of course, then that becomes a barrier to new people entering it and stuff. But seeing like the query builders, these UIs that help you build those queries, even the drill down apps, which just don't, you don't even have to look at the queries at all. And the assistant, new assistant, AI assistant that we're building, another way for people to interact with their data, and, you know, in natural language, suddenly, everybody can ask it, start to ask questions of their observability data. So yeah, I feel like that's also a thing. I see it, it just gets easier and easier to use over time, as well, which I think is great.

**Tom Wilkie:** So speaking of improving Grafana over time, what are the improvements in Grafana 12 in our, in our 12th release?

**Ryan McKinley:** \[16:08\] There's a lot, and there's a lot under the surface, I would say, but for me, the most important, kind of foundational shift that we've got is really a look at our API structures and how we're going to improve things over time. So dashboards as code, we've talked about forever, we've tried it forever. We are now taking kind of a schema first approach, a consistent API to all of the things that you read and write out of Grafana. So dashboards being the key one.

So one of the main things is we have a new model for the dashboard API that allows you to write dashboards essentially as resources, the same way you're used to writing them in Kubernetes or other systems, declarative systems. And then that allows the thing I'm most excited about, was a little experimental, so just beware here. Is we have a Git sync project that's gone on for years, but has been, it's kind of hindered by not having a schema first approach. But this lets you essentially configure Grafana so that when reading and writing dashboards we're actually reading and writing them to external static repositories that can have your own version tracking and governance for how those get deployed out. So we've all built kind of flavors of this to make it work over time, but it's now letting us keep that, but continue to edit in the UI, continue to use all the tools so that you're not always writing them by hand or from code that you barely understand.

**Tom Wilkie:** I'm really excited about this feature. In particular, and I'm sure many of our listeners have experienced this themselves, like we've got some Grafana instances with like 100 copies of the same dashboard because you go in and you just want to tweak that one thing, but you don't want to break the dashboard, so you create a copy and you tweak it in your copy, right? The Git sync, Ryan, is that going to fix that?

**Ryan McKinley:** \[18:12\] So yeah, so I mean, again, everyone manages their stuff however they do, like we all have different ideas of how to best do it. So it's hard to apply governance. And one of the biggest reasons is you don't know, can I safely delete something? And if I did, how would I know it's there, like sure we have them in the database and you can go back and roll back, but that's really hard. With kind of a Git backing, there's so many tools that you know, you already know how to use, that you can put your key stuff under one branch, promote it. There's so many workflows that we can deal with.

So again, how do you do, like how would you safely do a massive upgrade? Like it's a little terrifying in our totally closed off flavor, but if I could dump everything into a Git repository, run some scripts and see how the whole thing works and safely roll back, like a much more confident and there's ability to just delete a bunch of stuff or archive it if we're not going to use it. Because we have these problems of like, people with millions of dashboards, but you realize, well, that's because either they had programs write thousands of them and really that could just be a variable. But then how do you know how to which ones to delete? Can you safely do it?

**Tom Wilkie:** This is also going to allow us to like integrate more into existing development practices. Like, you know, and dashboards as code is about using, you know, CI tools to check your dashboards, to lint them to make sure they meet your organizational standards, right?

**Ryan McKinley:** Yeah, exactly. And again, with this common metadata wrapper around all our APIs and a common system for adding schemas, this opens up one of the things many people struggle with is our Terraform providers. So now that they're based entirely on schemas, getting Terraform to play nicely with it is significantly easier and more robust. So with this, we've got a new Terraform provider that reads these new versioned APIs.

**Mat Ryer:** \[20:12\] Yeah, I'd like to be able to have a PR and have a discussion about changes to a dashboard and decide as a group, you know, for code and dashboards. Be great. And be able to see and preview what changes might look like before kind of committing to them.

**Tom Wilkie:** Oh, does it? Yeah, I do. To be able to see it visually, not just from, you know, JSON at code. Although, that's how Torkel reads dashboards. Just looks at the JSON, is that true? Because he loves the beauty for UI.

**Torkel Ödegaard:** Well, before Grafana, that's how you edited most dashboards.

**Tom Wilkie:** Yeah, right. Dashboarding systems, what's the essence of JSON editor?

**Torkel Ödegaard:** No, I'm excited about dashboards, let's go.

**Ryan McKinley:** We've been, you know, as you kind of alluded to, Ryan, there's been lots of ways of achieving similar effects in the past, but they've all kind, you know, there's been, there's been 10 different ways of uploading your dashboards to Grafana, right? Like there's been, you know, 20 different formats for storing your dashboards in Git.

**Tom Wilkie:** And so like, I also like the fact that we're starting to get a little bit opinionated about this and start to say, no, this is the way you should do it. And this is the way, you know, we're going to make it easy. We're going to make it so that, you know, 80, 90% of the community does it one way and we can develop more, you know, automation and ease of use features on top of the one true way of doing it.

**Ryan McKinley:** \[22:13\] Yeah, and importantly, I think when people choose kind of automation and governance and review, it has meant that you have to work in tools that you're really uncomfortable with. And that's not why I use Grafana. Like I like to use the UI. I want to see what it is. I want to fuzz with it because I don't know what the stuff looks like. So the piece I'm most excited about is being able to merge those two into one workflow. Because you know, we see so many people that try with kind of programmatic rigor, but then once it's in the database, it's just too easy to edit and let's go with it. So how do we let you choose what balance of that you want is where I'm excited about it.

And in particular, then what that means for distribution models of like, how do I have a, you know, one central repository where we all think about it, but then I could deploy that out to multiple offline sites. How do I, you know, again, we have so many tools in the past to do this, but can we come up with a way where we're all sharing on our shared knowledge and improved procedures?

**Tom Wilkie:** One of the things you mentioned, like resource-based declarative like APIs and you mentioned the K word, right? Like, there's a little bit more to it than just a little bit like Kubernetes, isn't there?

**Ryan McKinley:** Well, so in its core, we've taken the foundation of Kubernetes is such a hard word because it brings so much baggage in what you think it is. So by no means is Grafana becoming Kubernetes. We're not saying you require Kubernetes. What we're using is the foundation of Kubernetes, which is something called an API server, which is essentially CRUD, like basic, you know, create, read, update, delete of resources with some additional kind of common metadata around them.

Kubernetes is complicated because it is both this core foundation and the application that we all kind of are aware of that operates pods, et cetera. So that's not what we're doing, but we are using API server framework from it, which one, you know, not invented here. So it's, you know, it has issues, but at least they're not the ones we made. It's something you can be familiar with. It should not be a surprise what our APIs look like.

And over time, we are migrating all of our current resources. So we're starting with dashboards and folders because they're really the core of it. We've had some for a while that no one really cares. It's like, oh, you can, you know, you can provision your playlists, yeah. Great, but, you know, dashboards is why this matters. Next up and soon are data sources, because obviously that's the next biggest one. But in general, it's all of the things that you edit in our UI, this process makes it so that we can have as code first and our UI is operating with those same patterns and resources.

**Tom Wilkie:** Actually, I think this is a little spoken about kind of industry trend that's emerging to start to adopt the Kubernetes API machinery like to power CRUD apps, right? Like, because the more I speak to other companies, other open source projects, the more I find they're doing the same, they're just not talking about it, right?

And the way I've seen this is like, this is really REST done right. You know, REST promised a lot of the things you just said, but there was never actually like a standard for what REST meant, right? It was never actually interoperable with anything else, right? And there was so many different flavors of it. You know, so my view is that the Kubernetes API machinery is just REST done correctly. Like, you know, what do you think?

**Ryan McKinley:** It's challenging, I'm going to say, because at small scale, 100%. Like the Kubernetes pattern of you have declarative resources and then a bunch of things watching them to try to make sure they eventually get into the state that you want is great. That does get hard when you start adding things like authorization to your permissions rules. Like, it does get really hard when you're modeling some of these things like secure values.

Like there's no kind of native Kubernetes way of doing that. So that's been our biggest challenges. Like how do we keep our robust authorization model? How do we keep, you know, real security with secure values? MD5 encrypted secrets is not, you know, a real secure value. So I agree with it in principle. It's the scale and kind of choices that Kubernetes APIs are built around the Kubernetes scales. And across Grafana, we see such a wide scale of usage that it's challenging.

**Tom Wilkie:** What do you, what do you mean by that? Like a wide scale of usage.

**Ryan McKinley:** You're not operating millions of pods, but people have millions of dashboards. And, you know, it's a few, few people, but the choices that you make around how to build things, the scale does matter.

**Tom Wilkie:** This is like, I think you're referring to like, you know, not being able to like have a secondary index on dashboards in the Kubernetes world, right? Is that the kind of thing you're talking about?

**Ryan McKinley:** I think it's buying into the Kubernetes ecosystem. Super powerful, but the way people write what in the operators are the, you know, conceptually one of the most powerful things about it. Is you just watching the resources you have. Operators written poorly just destroy databases, right? That's just how it, how they work. So you can get, you can very quickly get things to have cyclic updates. And when the numbers are small, that's no problem. When the numbers are large, it becomes a problem.

So that's been our challenge is how do we actually offer a platform that's Kubernetes based, but has enough guardrails so we don't, can keep things as resilient and strong as they are. Because at some level, you know, Grafana is just a repository for a bunch of static dashboards. So it shouldn't be too challenging to run, but with loops and stuff, things get complicated. I may be a bit in the weeds and sort of focus on these because these are my daily concerns, but the end result and where we're targeting is that kind of same experience of watch resources, any of them, and put the system in whatever state you want.

**Tom Wilkie:** \[28:26\] Yeah, when you talk about this, dashboards API is now having real schemas and things like this. What does that mean? What was it before? How did it not have a schema before? Surely it had fields and values and building on top of this API server means so many things.

**Ryan McKinley:** At the core, what I see is it's a versioned API is that the structure enforces that you have, essentially slices of your entire API that have versions to them. And what that means is I can change my versions over time. And Grafana, for whatever reason, we started off with one version and added fields to it and would change them, but it was never based on something with a strict API that came first. So we would always kind of retrofit our APIs, but without versions, you can't change them in meaningful ways.

So one of the more interesting things that I'm really excited about is dashboards have existed kind of in their form for such a long time. And there's so many out there that we can't really change it. But now that we're introducing a new API, we've taken the opportunity of like, oh, we could actually totally rethink this and address many of the challenges and limitations with them. So I don't know, Torkel, do you want to talk about some of the new conditional rendering?

**Torkel Ödegaard:** Yeah, absolutely. I think the first is to kind of go back to the dashboards has always had schemas, like since version one. Like we had a concept of a schema version and it's the thing that's been made as possible for Grafana to be backwards compatible. Like we make schema changes almost every version and we have migrations that sort of always roll forward. So you can sort of almost import a dashboard from 10 years ago or any sort of year, basically. And most of like they look almost identical and everything should work in like 99% of the time importing a very old dashboard works. Thanks to this kind of schema version.

The problem is that this was all confined to front end and the schema was not sort of the schema themselves versioned and that you have to go back to the source code to look at sort of the state of it and find the migration for that particular version. But the way that the schema was defined was mostly as the code. And so we couldn't actually translate the schema to different languages to an OpenAPI spec or to other formats or use them as a foundation to build kind of dashboard as code.

The schemas that have existed forever only mattered if you were writing the dashboard code itself but for all of the people external to that, there's been kind of only the most basic of reverse engineered traditional schemas, I would say. And that's the piece that's changing here is that we're making all of these resources start with something that is intended to be used across the whole ecosystem not just in our rendering engine. Given the fact that we have this new API

**Torkel Ödegaard:** \[30:01\] And we have these new schemas. We did rethink the schema quite a bit in terms of specifically how we store dashboards and how we store dashboard layouts. And the reason we did this was to enable us to introduce new layouts, new ways to structure dashboards. And this is something that I've been wanting to do for many, many years because the current or like the old dashboard layout engine was very limiting. Well, it's very flexible in a sense that it allows you to create very powerful panel layouts where every panel has its own position and size, which creates, you can create very complex panel layouts. But because every panel has a specific position and size, it makes dynamic features very hard. It makes it sort of, that makes it hard to do panels that can conditionally be shown or hidden because if you hide a panel, the positions of all the other panels will actually change. And it also makes the as-code kind of, you know, a little bit more painful because you have to sort of position panels with these absolute coordinates or the grid coordinates, instead of just caring about that. Yeah, I want just these three panels. They should be this size. But I don't care about their position on the screen or their Y position or X position. I just want these three panels and they are this size. So that's kind of having a new way to lay out panels that are more based on constraints. Like this panel should not be smaller than this or I want a layout that has sort of maximum, minimum one column or maximum three columns. And just think about more constraints and because a system built around constraints is much easier to make dynamic, responsive on smaller screens, responsive to conditional rules where we can hide panels for example, based on if there's no data available. That's a very popular feature request going back many, many years is like users want to hide panels that have no data or that can be evolved to more advanced conditions where you can hide panels if the data is above or below a specific value or if you have a template variable that hasn't been set. So things like that, being able to hide a panel depending on if a template variable is set or not enables you to create kind of more dynamic dashboard drill downs. Just clicking on something in a table, for example, sets a template variable and now you can sort of conditionally show a drill down view or a detailed view depending on what you clicked. Being able to do that without sort of creating a separate dynamic drill down dashboard, which is kind of how users do this in the past, or have that drill down view always visible but just show blank data or empty data. So anyway, conditional panels opens up a lot of new ways to build dashboards and new workflows. So that's something that I'm excited about.

**Ryan McKinley:** It's so many applications. Like, dashboards get you pretty far but then we find things that want kind of a stronger experience and you end up having to write a React app that essentially gives you a drill down into something where you know, like choose these three buttons and I'll show you four different layouts but an incredible amount of actively written and maintained code. And these new layouts really let us do that with our kind of more traditional WYSIWYG editor style. So I'm pretty excited about how much of the code we can get rid of based on these static declarative models that again share all of our experience and make these common experiences much more declarative.

**Torkel Ödegaard:** Yeah, and it's not a thing that we see in kind of many of the new data apps that we have in cloud and customers building using kind of the scenes framework. We see a lot of sort of use of tabs. It's not just kind of breaking up a dashboard like data experience into rows which is the only thing that we've had in dashboards. Many of the new scene based apps have used tabs too. So just break up a piece of kind of observability data exploration into more bite size chunks and that's something that makes a ton of sense to have in dashboards as well. So yes, a different way to structure it and make it more discoverable because rows can be below the fold, you have to scroll to see them and makes it a little bit more harder to take in what is available. So yeah, tabs is also a big thing that's coming to dashboards with these new layouts. And giving all these kind of new capabilities, we also wanted to improve the editing experience. So there's also a new way to select objects in the dashboard, edit properties that you can now sort of, now I can just correctly select a panel or a layout or a template variable and edit its properties without having to go into a full kind of full view or a modal or sort of the old settings, full page settings views. Now we have a little bit more kind of familiar editing experience if you have used other tools that have more of a blank canvas where you can place objects, select them, edit options.

That's going to be a little bit more the experience going forward. But let's be clear that this new editor, the advanced dynamic layouts, these are all, they're introduced as a kind of preview feature in Grafana 12 so you can see them. We really want people to check them out to see how well it works, but you have to know that it's experimental and that's evolving.

**Ryan McKinley:** Yeah, very important is that in Grafana 12, all these new dashboard layout features, it's highly experimental and it's not something that we recommend you enable in production. For multiple reasons, both for the state of it, it wasn't ready and it's also built on this new API and schema machinery that is also not at the production level state we wanted. To be clear, the dashboards now have multiple versions. The V1 of this is essentially the API you're used to seeing and it is production ready. You can use it today. We definitely encourage you if you're writing tools from code, look at these new APIs because this is where everything is moving. They should be familiar. The new layouts is our V2 of dashboards and in the same API versioning standard, it's the same dashboard is programmatically, on our behind our API, converted from one version to the other. So this is where we're going and it allows us again to make big changes when we need them. So we're not going to change the dashboard schema often. Like, I think Grafana has been around for 15 years, I think they're at three versions. And that's kind of the scale that we're thinking about. It's like we've had one version that's gotten us pretty far. It will look very similar to what it is. We're introducing a new one now that allows this nesting, has the schema first approach where panel schemas are clearly separated out from the dashboard. But this is where we're going right now. And by virtue of using that, we can enable all these other tools like GitSync and external provisioning. But there are a bunch of things that are actually pretty real in 12. So let's...

**Tom Wilkie:** Yeah, so we've talked about the dashboards as code and APIs. We've talked about the new dashboard rendering but there are a bunch of really like much smaller incremental improvements in 12, right? Like that we're equally excited about. So let me get one of them actually, like question for you, Mat, because I think you worked a bit on this, didn't you? Like you worked on the drill down apps?

**Mat Ryer:** Yeah, yeah. For anyone that doesn't know about the drill down apps, this started with, I think, Torkel you did a kind of prototype data trails it was called, which I still think is a really cool name. By the way, it never got used. But this was the idea of, instead of you having to write the query, we'll practically just write lots of those queries and show you the results and you can kind of just browse through them and you see something at a high level which you can then drill into and let you go through the details to figure it out. And this turned out to be very popular. And again, it's part of that usability. It makes it even easier now to use and therefore more people can use it, you know, a developer who might not be in Grafana every day and engineer still might want to be able to go and look at some bits sometimes. And before they'd have to have learned PromQL, LogQL and various query languages and now they don't with the drill down apps. And yeah, the team, the team are just kind of phenomenal though, there's four squads are basically building each app, just a bit of internal there. But the, yeah, the metrics one, I think, phenomenal, it's got new filtering and grouping. A big problem in metrics is there's just so many metrics. You know, you don't have to have a massive system to still have lots of metrics and finding the right ones is a big challenge. You get this sea of graphs. So being able to just do simple filters like prefix and suffix filtering which you can now do, very useful and grouping by labels. So yeah, if you're using labels properly, that's an extremely powerful way to be able to drill into metrics and stuff. So very excited about that. And I know they're going to be doing more work. The logs drill down one has a new JSON panel which I think is quite exciting. And by the way, we did a Grafana Campfire call on this. If you search your favorite search engine for that, it's a regular community call we do. There's one on the drill down apps where we drill down into these apps, which is a bit meta. But the JSON panel is great. If people use JSON logs, now you can see the fields properly. You can choose which fields you care about. And it's just UI improvements phenomenal there. And then profiles and traces, they're also GA. The traces has this really cool auto sort of investigation tools. They just give you tools right in the front end for you to just do sort of common things like looking for root cause correlation, comparing things, highlighting differences and stuff like this. So yeah, and they're great. If you haven't used them, they're open source. You can just, I think they ship now with Grafana. So yeah, drill down. I've actually started, you know, I wrote PromQL for almost 10 years, right? But like even I start using the metrics drill down app now more than I end up writing PromQL and I find it faster and easier and quicker. And ditto with, you know, I helped design LogQL as a language and I still find the logs drill down app is easier. So yeah, like it. They definitely get my massive seal of approval. Like I use it every day.

**Tom Wilkie:** They're good because you can start there. You can still always, at every point, you can open it in Explore. So you can always go back to the more advanced and you get the query, you see the query that it's built for you.

So it's actually a good way to learn as well if you are interested in that. But yeah, the usage has been great. Like they've been really, they are really popular and I think it is, yeah, because just easier to use and you do less and people are busy.

**Torkel Ödegaard:** We've also got some new themes, haven't we?

**Tom Wilkie:** Yeah, well, we do have some new kind of look and feel experimental themes. But I think it's still, I think, they are more just experimental but they are enabled as an embedded default. And these are, you know, maybe not sort of themes that will stay around in their current form forever. They are more like trying to explore what Grafana could look like.

**Mat Ryer:** To revive and win.

**Tom Wilkie:** To revive and win. Yeah. But because Grafana's had a kind of a theming architecture for a number of years but we've still just had these like two dark and light themes. So I was really happy to get this into Grafana 12 to finally have sort of, yeah, explore some new ways where Grafana can look a little bit differently, have some slightly different sort of color palettes. I think these new themes don't include a new visualization color palette. I think they are mostly for the UI itself but I think that's also something we hope to look into in the future. But yeah, check them out. They are in the profile page or there's also accessible, I think, from the profile menu so you can change kind of theme anywhere you are. So I find the themes kind of fun. Just, it's nice to change it up a bit but it's also been really interesting to see how we can use this to audit our own UI and in particular plugins because changing the themes makes it really clear which ones are using the standard classes, using the standard libraries versus have kind of shoved in, you know, raw CSS forever which doesn't work well as we go. Same thing with internationalization. So there's been a huge amount of effort towards adding internationalization places and again, we've got some special languages you can shift to that really make it clear when you have or haven't translated or enabled this across our UI.

**Ryan McKinley:** The way I do it is the way I validate that my themes and all my code is correct. So I send it to Torkel and then he'll just send back and say, this border should be border weak. This one, the border radius is non-standard. He just sees it all. It's like Rain Man for Grafana dashboards.

**Torkel Ödegaard:** I still run, I still run my Grafana in Light Mode.

**Tom Wilkie:** Our manager would do. Yeah, I much prefer Light Mode to Dark Mode. I don't know why. I think Dark Mode looks a bit 80s.

**Torkel Ödegaard:** Well, actually, I like it. I also heavily use Light Mode that kind of depends on time of day. But like, if I'm coding or working late at night, I might switch to the dark theme. Mainly because I haven't for a couple of years I haven't been super happy with the dark theme. There are things that, I think the new themes, one of the new themes at least tries to solve some of them. But yeah, I actually like the light theme as well for at least some. And for screenshots, you know, embedding graphs in a Google Doc that always ends up light theme.

**Ryan McKinley:** Yeah. No, definitely. You've seen me there. That's basically my day job now.

**Tom Wilkie:** Yeah, it is. We've also built a bunch more data sources, I think, haven't we? Like that 200 number keeps going up.

**Torkel Ödegaard:** Yeah, it's impressive to see these things just keep appearing. I mean, obviously we internal Grafana support a bunch of them, but we keep seeing new contributions for stuff I haven't seen before. But some of the ones I've seen recently are Drone, Netify and Zendesk. I think behind the scenes, I'm not an expert on this though, but I think the data sources team in Grafana have like a new, like more declarative SDK for building these data sources, which is allowing us to produce more, more quickly, with a lower maintenance burden. Does anyone know anything about this? Sounds clever.

**Ryan McKinley:** Yeah, so most of the data sources are translating other people's APIs into the data response that Grafana UI expects. And so coming up with more declarative ways of doing these API transformations, which written in Python become pretty obvious, like that's what Python thrives at. But over time, we've come up with good ways of doing that within the ecosystem that we're building with.

**Tom Wilkie:** Nice, and on the kind of topic of data sources, one of the things I'm, this is actually for me is one of the things I'm almost most excited about in the Grafana, in the recent Grafana releases, is our new SQL expressions. Like, do you want to, Ryan, I think you've got a bit of history here, haven't you?

**Ryan McKinley:** Oh, this thing is great. It's essentially, you know, the single pane of glass story has always been that your data's in multiple places. Like, they all have various levels of capabilities. Yes, if it's all in Prometheus, you could probably come up with some PromQL expression to do most of it. But then once you start trying to use it for, you know, business data, it gets awkward. And we have built transformations, we've had expressions over the time. But in 12, we've introduced a new kind of way to combine results from multiple queries and then execute them as if they were SQL joined tables. So all of the features that you get from essentially a MySQL query, you can use as if each of the results of a query was one of the tables you're joining with. And this really greatly improves how much we can support and how common that can be.

**Tom Wilkie:** \[48:20\] Yeah, I really like, I mean, also the comment you made earlier about like AI knows how to write SQL. It doesn't know how to write 200 different query languages. Like, and so like, I'm really excited about a future where we can do more with this SQL engine and where we can use it, you know, to join across data sources and even maybe one day directly query data sources. I think that'd be pretty cool.

**Ryan McKinley:** Yeah, we recently added support into the assistant for a lot of SQL stuff. So yeah, and it is good at that. It's quite good at PromQL. But you know, because so much of Grafana is open source and so much of the conversation in the ecosystem is open source, LLMs have a good intuition about Grafana in general. So if you, even if you're using ChatGPT to ask it about Grafana questions, then it does quite a good job because of that.

**Mat Ryer:** And other LLMs do exist.

**Ryan McKinley:** Other LLMs do exist.

**Mat Ryer:** Anthropic, we use that one. I've got Claude at home. Claude desktop, because I just like desktop apps. And I was doing some SQL yesterday actually. It generated some really interesting structure of my SQL, like I would never have done myself, like it's creative.

**Tom Wilkie:** So this SQL expression started as a hackathon project. I can remember a very early hackathon use. So we're using Dolt Hub's. They have a MySQL engine written in Go. And four years ago, I think it was four years ago in the very first hackathon. I used the same SQL engine to do a project called PromSQL, which was to query Prometheus with SQL. And I remember showing it to Sam who ended up doing the SQL expressions and replacing. Am I right? We were using DuckDB as a sub-process. And then we moved to Dolt Hub's Go. Correct?

**Ryan McKinley:** Correct. So yeah, one of the hackathons looked at pumping data into DuckDB. Super powerful. But with the Dolt Hub flavor, we don't even have to do the translation, because it's already natively in the formats that we have.

So yeah, I'm a big fan of the Dolt Hub Go MySQL engine. It's impressive. It is a very impressive piece of code as well.

**Tom Wilkie:** Yeah. And an incredibly well-supported community, friendly community, very helpful.

So continuing through the list of small improvements in, well not small, some of these are pretty major improvements in Grafana 12. Right, we've also got some new authentication work that I will say I don't really understand. Like, what is SCIM?

**Torkel Ödegaard:** SCIM, I think it's a standard or a term for syncing sort of users and groups and permissions, where we've had some kind of external, we have had support for LDAP and other authentication systems for a while. But we haven't had sort of support for active syncing, where an external source of users and teams and groups can actually be pushed to Grafana. We've had some sort of syncing that happens at sign in. And especially for teams, this has kind of been really painful because you have to sort of still kind of set up the teams and then map them to an external team or if it's an LDAP team or something. Anyways, it required a lot of setup. And to actually then kind of set some permissions, you have to sort of wait for user to sign in for them to show up in Grafana. So this SCIM kind of authentication or sort of syncing system, it's much more easy and active and you get all your teams and users show up in Grafana right away for you to be able to assign permissions. It stands for System for Cross-domain Identity Management.

**Tom Wilkie:** Yeah, that was definitely a background in there.

**Mat Ryer:** Yeah. Yeah, because you wouldn't normally say a SCIM or something, dude, it's not a pattern.

**Tom Wilkie:** Sounds like, yeah.

**Mat Ryer:** \[52:25\] But still, that's cool though. So this is good, you don't have to go in, mess around with all those. And this is an enterprise feature, right?

**Torkel Ödegaard:** This is an enterprise feature.

**Ryan McKinley:** Is it an enterprise? It's one of those sleeper ones that I think was just one of many small things we've built. But this is the one that we're hearing the most about of how big a change it's making for people operating their systems.

**Tom Wilkie:** And last but not least, we've mentioned the AI assistant a couple of times, but we now have a new way of plugging in things into the side of Grafana. What's this about?

**Torkel Ödegaard:** Yes, so as part of the Grafana 12 and the GrafanaCon keynote, we gave a sneak peek at our new Grafana assistant, which is only available on Cloud and something we actually launched now in, I think, sort of a preview form on Cloud. It's probably been, you know, yeah. So this needed a new kind of plugin extensibility, kind of feature in Grafana, where a plugin, because this is implemented as a plugin, where a plugin could actually take over a sidebar pane no matter what page you're on. So yeah, that's pretty cool too. Because I think coming out soon is a new documentation and kind of learning assistant that will come through the same sidebar context. So we hope to...

**Tom Wilkie:** So this is pluggable, and like people could, third parties could also plug themselves in there?

**Torkel Ödegaard:** Yeah, that's the plan. That's the plan. I think there's no, I think, official plugin docs for this type of extension, I think, is what we're calling it.

**Ryan McKinley:** Extension, yeah. It's actually some new type of plugin. So any sort of data source or panel plugin can actually register themselves as kind of using this new extensibility. So it's not a new type of plugin, it's just a new thing that a plugin can do.

**Torkel Ödegaard:** \[54:28\] And it's sort of basically, I think, any type of plugin.

**Ryan McKinley:** Yeah, and this was actually, this took a long time to do. This was hard to get in, and I really sort of appreciate the friction that we had, because we were kind of really pushing to get this sidebar in. The amount of care that the team take on just looking after that user experience, not wanting to just throw everything into the UI, it's really hard to get these kind of changes to happen. And I think that it should be, like people are really sort of paying attention to it and making sure it's going to fit alongside other workflows and things. So yeah, it was kind of, and it's great. And I think when you, if you've used it, if you haven't used it, go and try it. But it's just such a nice implementation. And technically, we went through a few iterations in the background as well to get it right. But that was a kind of a healthy friction, I felt like, because, yeah, you don't want anyone to just be able to just add all sorts of loads of features. We'll end up with the Homer Simpson car.

**Tom Wilkie:** So this is all lots of very exciting things. But can you tell us anything, sneak peeks about what's next, what's coming up? Can people just go to the repo, by the way, and just find out all themselves? Can they just, because it's open source?

**Torkel Ödegaard:** It's all open source, so yeah, you could see. But I mean, I feel like we've hinted at it already. A lot of what we're doing has been really laying the foundation for what we're building. So we have our version one of Dashboards API that essentially maps to what you've already seen. Clearly the new schema for version two and for how we will have declarative resources for all of the objects that you manipulate within Grafana is really the stuff I'm most excited about and that work just continues along the train of migrating our bespoke APIs into a common structure. So that's the biggest thing occupying our teams at the moment.

**Ryan McKinley:** Yeah, and then for my perspective, like the future, the short-term future is actually delivering the new dashboard features that I talked about, because in Grafana 12, they weren't really ready for public preview. So working towards making them ready for public preview is my main kind of goal. What's kind of beyond that, I have actually not sort of yet decided on, at least for me. Like I think there are multiple things that interest me where I could see myself thinking more about how AI will fit into building dashboards or other workflows. And another thing that I, which I also hinted at, is where do we take the new themes or the look and feel of Grafana? I think Grafana has, in general, kind of looked very similar for the last eight years. How do we push that to the next level and these new things given AI and these new kind of assistants that could warrant a rethink for the rest of the UI in terms of how we structure it and how we think about workflows. But yeah, it's a little bit of an open field there, in terms of, at least for me, what's the next beyond?

**Torkel Ödegaard:** I'm still thrilled to get back to a bit of how do we build operational dashboards, given a lot of our new abilities with layouts and how we render things. We're like much better positioned for this mix of pure observability, but also a mix of like, how do we also operate on the systems that we could work with? Obviously done with care, but.

**Ryan McKinley:** And I think another thing that is big in terms of Grafana as a whole is work towards true multi-tenancy and breaking it up and making it easier to operate at scale. And that's at least something we care a lot about, but I think it's going to have impact for open source users as well, potentially make Grafana a little bit more composable for different scenarios.

**Tom Wilkie:** Yeah, we're running tens of thousands of Grafana instances in Grafana Cloud and moving that towards a more scalable like multi-tenant microservices architecture is our long-term goal. I think like we could do a whole episode just on how we're doing that and, you know, maybe less relevant to the community, but yeah.

**Torkel Ödegaard:** I suspect we'll do a few.

**Mat Ryer:** Well, you can check out all the stuff that's new in the documentation, if you just search for, it's easiest, what's new in Grafana. The docs are there and they do a great job, I don't know who does this, but the release notes and everything that you get, they get a proper blog post with videos and all sorts every time. It's like, yeah, they do it really well. And the keynote from GrafanaCon as well. We did, you know, GrafanaCon in Seattle a few months ago was all about the Grafana 12 release and, you know, there's loads of videos about this on YouTube if people are interested.

Okay, well, that's all the time we have. Thank you so much for joining us on Grafana's Big Tent. Tom, always a pleasure.

**Tom Wilkie:** You too, Mat. Thank you.

**Torkel Ödegaard:** Yeah, thank you.

**Ryan McKinley:** See you for breakfast tomorrow.

**Mat Ryer:** Oh, yeah, forgot about that. Do you want to go to Dishoom for breakfast tomorrow?

**Tom Wilkie:** Why not?

**Mat Ryer:** \[60:00\] Yeah, why not? Okay. Ryan, Torkel, you want to come for breakfast? No, it's a bit far. It's a little far from Torkel. Don't worry about making it flowery or polite. Yeah, just say what you really think, Torkel. Straight out, doesn't want to go.

**Torkel Ödegaard:** No, thank you.

**Mat Ryer:** That's basically the kind of review my code gets. I'm not sure, Torkel. No. No. No. It's very helpful, though, to be fair. Okay, thank you very much, Ryan, Torkel, Tom, and thanks for listening. See you next time on Grafana's Big Tent.