Full transcript

Introduction

Phil Ewels: Hello and welcome to the Nextflow Podcast. You are joining us for episode number 58, and today’s episode is all about Nextflow plugins. This is actually plugins podcast number two, because we did one way back in April, 2024, episode 35. Ben and I sat down and talked all about plugins and he actually led me through building a plugin live in the podcast.

And that episode of the Nextflow Podcast was really popular because plugin documentation has been a little bit patchy, and so people have used that as a reference. But things have moved on so much since then. And I’m happy to say there’s much more material about how to write plugins these days. We thought it was time for a new episode all about plugins.

Once again, I don’t actually know that much about plugins myself when it gets down to brass tacks, so I’ve brought some guests in to teach me all about it and tell us all about the new updates.

Welcome to our guests

Phil Ewels: Everyone knows Ben by this point, if you’ve been listening to the Nextflow Podcast. Thanks for joining us. Ben’s an engineer at Seqera working on the Nextflow team, and then we’ve got two special guests in addition. Jon and Rike, do you wanna say hi, guys?

Jon Manning: I’m Jon Manning. I’m a bioinformatics engineer. I’m also learning about plugins as I go along and, to do something, I’ve actually had to write some material on this recently for our training program. So, I’m happy to be here.

Rike Hanssen: I’m also bioinformatics engineer here, and I, a couple weeks ago, went through Jon’s training to write my first plugin. Happy to be here to talk about it.

Phil Ewels: You’ve got all that training material fresh in your mind. Alright, Jon, as the new author of the new plugin training material, you should be a pretty authoritative figure on this topic. Can you take us into an introduction as to what plugins are?

An introduction to plugins

Jon Manning: Plugins are really just a way of taking your own magic, and sprinkling it, into Nextflow without ever having to really go anywhere near the Nextflow code base itself. The common use case for this is just providing your own functions. For example, you, people often find themselves writing functions, which might be useful across different pipelines. I’m sure we’ll talk about some of those as we go through today. And a plugin is a way of doing that. So you don’t have to constantly redefine those Groovy functions. And also, if you want to notify yourself when certain things happen in a workflow, you can use plugins to, to provide that kind of functionality for you.

But in kind of outline, when you write a Nextflow plugin, you end up with a file structure that looks something like this. We have some tooling to help you with this. Don’t be too scared by this layout. But you end up with a bunch of files that look something like that, a bunch of Groovy files.

And in the course of developing your plugin, you’ll go through and edit those files and add in your own bells and whistles as you go along.

There’s a bunch of tooling that’s underlies how you do that. For example, this Gradle functionality, which, as a non-Java programmer, was a bit strange to me. But it’s actually not that hard to use to, to build your plugin and get things going.

Phil Ewels: You’re taking us through the Nextflow docs here and the files you’re talking about. These are the, basically are scaffolding files that are created when you start building the plugin, right?

Jon Manning: Yeah, exactly.

Phil Ewels: So you never start from a completely blank page as such. You start off a starter plugin basically, and then modify.

Jon Manning: Yeah. And I think maybe we’ll get to this in a second, but one of the cool new things in Nextflow is this kind of ability to quickly scaffold a plugin project for you to work with.

And once you’ve built your kind of components and your functions or your observers, there’s a whole bunch of configuration functionality you can provide in this doc information material about how to do that. So you can customize the behavior of how your functions or your observers work to add extra functionality as you go along.

And one of the, one of the even cooler things you can do, but slightly more involved and, but not something that I’ve done a lot, is actually you can actually build your own executors. So interact with novel compute systems, for example, and or novel and also novel file systems. But that’s just the outline of what a plugin is. You can do all those different things and you work with these files to do it.

Phil Ewels: So, from a point of view of someone who’s running Nextflow pipelines or writing their own Nextflow pipelines, I guess many of us will have seen plugin include statements. How common is it for people to write a plugin, which is going to be consumed by many different people, versus writing custom plugins just for themselves or for their company?

Jon Manning: I think probably things are changing quite quickly in that regard. I would say, I think probably because this tooling has just, has fairly recent and used and the ease of doing this is fairly recent, it’s, I suspect that the situation will change and there’ll be more and more of these kind of widely used community plugins.

I, I guess there’s probably a lot of hidden iceberg kind of situations where companies are using them behind the scenes that we dunno anything about. But yeah, I hope that we’ll see a lot more community plugins as we move forward. And this becomes more, more or simpler for people to do themselves.

Phil Ewels: And you can write custom Groovy in your Nextflow pipeline, right? You can.

Jon Manning: Yep.

Phil Ewels: You could write it pretty much anywhere, but now it’s in the lib directory. At what point does it make sense to flip over into a plugin instead of writing within your pipeline?

Jon Manning: I always refer back to the, because it, Larry Wall had that, the print, the three attributes of a, of, oh, sorry, I’m, I’ve forgotten what the reference is now. But basically I’m lazy is what I was getting to with that, with that discussion. So, as soon as I start having to copy and paste code repeatedly between things, that’s the point at which I wanna factor it out and make it a plugin and import it.

The one very, very important example is nf-schema, which is widely used across the nf-core community and used to read in sample sheets. And that’s something we do across pipelines repeatedly. And so you wouldn’t wanna be, have to be reimplementing that parsing and schema validation across pipelines.

Yeah, it’s just, be lazy. Don’t reimplement the same thing again and again, if you can help it.

Phil Ewels: Nice. And we have plugins for other systems as well, right? I mean, a second ago before we were on camera, we were talking about nf-test. And nf-test has its own plugin. These are totally different things, right?

Jon Manning: Yeah, you’ll come across these little plugins that nf-test has. There’s an nft-utils plugin that, that is used widely across nf-core as well to provide little nf-test help. But that’s a different thing.

Phil Ewels: Okay. So you mentioned utils plugins for common functions. You mentioned that you can go as far as building custom executors and I guess Nextflow itself, many of the core executors are actually built as plugins within Nextflow, right? That’s how the code base is organized, so that makes sense. Is there any other kind of common use cases you’ve seen come up where people have built plugins to solve problems?

Common plugin use cases

Jon Manning: I mean, the observer thing that I mentioned in the intro is a very common thing, right? So to make certain things happen when certain events occur in the pipeline. So there’s a plugin for calculating the carbon footprint of pipelines, for example. And that plugs into certain events in the pipeline. It notices when certain things happen that would trigger carbon energy usage. And it’s, it is the ability to be able to hook into those events that allows that plugin to make those calculations and give those results to you. And you can use that same kind of functionality for talking to Slack, for example, to send events off to Slack when certain things happen or to talk to your local database to track runs, for example.

So it’s that hooking into events and responding is also a really useful feature of plugins.

Phil Ewels: Nice. I know we’ve been working on a Slack plugin recently, I think. Rike, I’ve seen you, you doing some stuff on the nf-core side with a Slack plugin. Is that right?

Rike Hanssen: Yeah. Adam recently implemented it so we get better notifications when the full-size tests fail on nf-core. So when we release an nf-core pipeline, we like to run these big tests on AWS to make sure like the pipelines live under real-world condition, so to say, and we need some sort of notification system.

We used to have this nf-core for a while and some older implementations of talking to the Slack API was way outdated. Nobody was updating it anymore. So now we have this nf-slack plugin that on a regular basis is annoying people on various Slack channels if their pipeline starts failing. So hopefully it gets rolled out with the template release that is coming in a couple of weeks. And then everybody gets notified in the individual Slack channels for their respective pipeline on successes or failure. So hopefully lots of successes, but it hopefully also makes it easier to track failures and just makes it a little less clicking and a little less searching for the pipeline authors and just have the message be brought to you instead of looking for it.

Phil Ewels: Brilliant. Yeah. So that’s v4 of nf-core/tools, I think it will be. Very cool. Ben, you and I chatted back in 2024 about plugins and many of these core concepts were the same back then that they are now. I think even my dummy pipeline might have used some kind of a event observers and things. Changed? Can you take us through some of the new stuff that we’ve rolled out in Nextflow since then?

What’s new in Nextflow plugins since 2024

Ben Sherman: Well, Jon gave a great explanation, highlighting the most important things you can do with a plugin. Those, and you’re right, those haven’t changed at all. You’ve got the trace observer, custom functions. You can write custom executors, custom file systems. Although those are a bit more advanced, but we’ve seen a few people try their hand at those. The main things that have changed have been the tooling around all of that. So, Jon showed an example of the plugin scaffolding. This part has been simplified significantly. It used to be that we just had a sort of a example plugin called nf-hello. I think it’s still out there. I think we just archived it. But it was the plugin development process was take this nf-hello it, rename it, rename all the things in it however you want. And then go from there. And you had to figure things out on your own. You had to study even a Nextflow code base itself to understand how Nextflow was structured, to understand how the plugin was structured. Then finally, when you were ready to release a plugin, you had to build it, to figure out how to create a release, then you’d have to submit a PR to this giant JSON file in the Nextflow GitHub org and wait for Paolo or you or me or someone to approve it. And then it, then repeat that for every single new release. Even something like a bug fix.

What’s changed since then is we’ve tried to automate and streamline as much of that as we can. And so this starts with a new command in the Nextflow CLI called nextflow plugin create, where you run that and you supply the things that normally you would have to go in and rename yourself.

You just say, “Hey, the plugin is going to be called Slack, organization is whatever.” And then it generates that plugin scaffold for you, with everything already being named correctly. We try to hide as much of the Gradle and, and Java boilerplate behind the scenes as much as possible so that the only thing that you really see in your plugin template is the code that you’re going to work on, and a Gradle configuration where you can specify all the important things like what Nextflow version you wanna use, the different attributes of the plugin, things of that sort.

Phil Ewels: When we did this before, the nf-hello plugin had loads of files, which were very unfamiliar to me as a Python guy. And that’s one thing that struck me when I saw the new backend system for it. Much of that stuff’s been abstracted away now, which is much, much cleaner to work with.

Ben Sherman: I’m this way, and a lot of software engineers will tell you this. My favorite thing, honestly, is deleting code. And so updating these, every plugin that I maintain to use this new Gradle plugin, you just see all that red, deleting all of that boilerplate makes me very happy. So that’s one huge win. Next huge win is just the development process. So now it’s a lot easier just to build a plugin, also to even test it locally. There was a way to do this, but I was copying Makefile snippets and parsing ’em around to people. And now it’s built into the plugin template.

You can just say make install, and it will actually install your plugin locally. And then as long as you call it correctly, when you run it, you can just, you can test your plugin locally. You don’t have to wait to be released and do all of that. And then when you’re finally ready to make release again, that’s just another make command.

You do make release. Now you have to make an account on the plugin registry and get your API token and all of that. But once all your authentication is set up, it’s just a single command. And then you release it to the registry instead of making a PR or waiting for people to approve.

The only thing you have to get approval for is the plugin name itself. So you go onto the plugin registry, you claim a plugin name that you want, make sure nobody else has claimed it. And then Paolo or me, one of us comes and approve it. And then once you own that name, you can publish releases to that as much as you want. Obviously don’t go crazy. We haven’t had anybody test our API limits yet. But the plugin registry is this sort of replacement for GitHub. It’s this central repository for storing all these plugins, publishing them, also discovering them. There wasn’t really a website or frontend for just looking for new plugins. And now we have that. So you can go to registry.nextflow.io, then you can just search plugins and every plugin has a basic description of what it does. It’s got a link to the GitHub source code. We try to make sure that every plugin that gets published is open source there. So you can dig through all that same stuff.

So putting all that together, you end up with a much more streamlined process. And we saw the effects immediately as soon as we released all of this in Nextflow 25.10. I think every single person on our side of the team came up with some idea for a plugin just to do it, and they, there was a plethora of plugins published in the ensuing weeks. It’s just continued to go up, so it’s been a huge success.

Phil Ewels: Yeah, it’s going to, the only downside is it’s going to hurt my GitHub contribution metrics that I don’t have to merge these pull requests all the time to that massive JSON file. We’ve talked about the registry a little bit in, in previous podcasts and in webinars and Summit talks and stuff. So hopefully many people will be familiar with it. I should mention also that, that JSON registry isn’t completely dead yet. Is that right? When you published a plugin into the registry, it’s, if it’s an old version of Nextflow, it still gets backported. Is that right?

Ben Sherman: Yeah, so we still need to keep using that JSON index because all of the previous versions of Nextflow still use that, right? Nextflow 25.10, you know, moving forward, uses the plugin registry. But we need to keep that old JSON index alive, if nothing else, for just all of the plugins that already exist. We have this automation set up where if you’re publishing plugins that are intended to be used by older versions of Nextflow, like 25.04 or 24.10, they will still get pushed to this registry automatically. The plugin registry will make that PR for you. And I think currently Paolo still has to go in and merge all those PRs. I figure at some point he’ll get annoyed enough that he’ll just allow the auto-merge. But I think he just wants to be very careful that something weird doesn’t happen and you don’t wanna break the JSON. But in any case, you don’t have to worry about that anymore. If you publish a plugin and it’s something that should be on that JSON index, it’ll get pushed there and then everything just works.

Phil Ewels: And I want to also point out this point that if you say that the minimum Nextflow version is 25.10 or above, I think it is, then, then it won’t do that automatic sync, ’cause I had a couple of people asking me quite recently about why their plugin hadn’t shown up on that, on the plugins index yet. And I think that was the reason.

Ben Sherman: Right, because Nextflow 25.10, I think it still can use the old JSON index if you use the right environment variable. But it is designed to just use the plugin registry. If your plugin is intended for 25.10 or later, and there’s no need to publish it to that JSON index. It’s also just a way for us to control the size of that JSON file, especially now that the plugin registry is exploding. We’d rather not JSON file explode, because for older versions of Nextflow, they’re downloading the whole JSON file on every run. We aren’t particularly intelligent about managing that.

So putting a cutoff at 25.10 put a cap on the long-term size of that JSON file. Over time, as people, you know, are using 25.10 or later or so on, that index will eventually just stop growing. We may eventually freeze it, say like, no more updates to it. But obviously we’ll never take it down because it will always be needed by those older versions of Nextflow.

Phil Ewels: And are there any changes to people using Nextflow plugins around this? Or is this all completely behind the scenes?

Ben Sherman: If you’re just using plugins, I don’t think you’ll see any changes. That point of view, it’s just the same plugin config that you’ve always used. It’s just that under the hood, it might start pulling from the plugin registry instead of GitHub. But other than that, it works the same way.

Phil Ewels: Rike, you’ve gone through this process recently. Is Ben telling a truth? Was it simple?

Demo: Rike’s nf-emoji plugin

Rike Hanssen: I somehow had avoided learning about plugins for a long time because in my head I made it to be this big, complicated process that I don’t have time to learn right now. And I thought about nf-schema and that, that’s probably very complicated plugin. And then I went through Jon’s training and we’ve looked at the new documentation that Chris had written.

And it turned out to be really straightforward and fun to write. Compiling was very fast. You could just iterate quickly over new things and just start playing around with it. So yeah, confirm this. This really works pretty well now. And also the publishing was very straightforward. It was really a pleasant experience to develop it.

Phil Ewels: You work in very high-impact scientific domain. You reach into the internals of these really massive companies. I’m sure that whatever idea you had for a plugin must have been equally high-impact and important. Can you…

Rike Hanssen: It’s a really important high-impact plugin that prints random emojis when you run your pipeline and sprinkle in progress VARs made of waves or spaceships or horses that are racing. And if you’re really successful, in the end, it also prints a little bit of confetti to the terminal.

Phil Ewels: Love it.

Rike Hanssen: They’re really important things in scientific analysis.

Phil Ewels: Oh, you can’t go too far by sprinkling a little extra joy in people’s pipelines.

Rike Hanssen: To be…

Jon Manning: Well, we all learn.

Rike Hanssen: The QC that prints emojis sometimes for teaching, which is…

Phil Ewels: FastQE, right? Fast… instead of FastQC. I think there’s a MultiQC module sat in review to support FastQE. Been meaning to get round to adding out for ages. Were you going to say something, Jon?

Jon Manning: Oh, just that we all learn by play. It’s a great way to, to get your hands dirty actually on plugins is just to go and do something crazy. And I’ve not tried to publish it, but I made a plugin based on the popular British quiz show, Numberwang, which people outside of British scope won’t understand at all. But it’s just, is good to do something completely off the wall and just try, before you have to do something important.

Phil Ewels: Yeah. And if you get anything wrong in your code, you can just shout, “That’s Numberwang!” And everyone says it’s fine. Sorry. Too niche.

Jon Manning: Yeah, we’ll stop British in British jokes now.

Phil Ewels: Yeah. Anyway, Rike, can you tell us a little bit about your plugin? I know it’s, I know it’s only silly, but it’s a good educational tool, I think, in this context. Can you tell us how it works and show us something?

Rike Hanssen: Let me, for the people watching, let me just show you a little bit and I will talk my way through it. So, it’s this very small plugin that I developed with the new, um, “nextflow create plugins” command or whatever it was. But it’s doing a couple things. It’ll figure out what season you are in and then tell you about it.

I got a few days marked like Earth Day. We talked about the nf-co2footprint plugin recently, but also other important days like Pi Day or Valentine’s Day or Christmas will tell you a counter until when. And then you have a little progress bar that progressively fills up, and then it’ll tell you in different themes how the pipeline is doing.

So, for example, here we have a space theme, and it’ll tell you in the space emojis how many processes have succeeded, failed, or are cached. And yeah, when you, let me, for people watching, I can go ahead and run it. But we’ve got other themes in here such as, I think, horses, spring, Halloween, just various silly little themes. And then also a little confetti option that needs to be installed separately. When you implement it, you can set up like a new scope for your plugin. So I called mine the emoji scope and added in some parameters in here. We can go ahead and print, for example, the confetti on success if we still wanted to in terms of code changes.

So Jon briefly showed that you get this. There we go. Bunch of random signs here.

Phil Ewels: I was wondering what was going to happen with the confetti. I was like, have you, would you do that in a terminal? But that was beautiful. ASCII confetti. I don’t think I’ve ever seen that before.

Rike Hanssen: Yeah, it’s actually shelling out to another tool. Which also, I made it optional because maybe not everybody would want to install it, but it’s silly enough for me, when I run it locally.

Phil Ewels: Just before you move on, I also noticed that every time you run a demo command there, you were rerunning make assemble and make install each time, presumably ’cause that’s needed for your, for the config changes, or…

Rike Hanssen: So this is only needed when I actually make code changes. I could just do, now that the code changes are done, I could just do nextflow run workflow/main.nf. I am just, I think, muscle memory, in my history, I have it right now because I, in the beginning I didn’t quite, I needed to get used to always assemble of my code.

Like, can, when you write normal Nextflow pipelines, you don’t need to do this. So people that are maybe more used to programming in another language, so I might be more used to it, but I kept forgetting that I need to assemble and run all these make commands ahead of time.

Phil Ewels: The other thing I noticed is how fast it was, ’cause it wasn’t really holding you up at all.

Rike Hanssen: Now it’s, I barely notice it. The other thing that’s really nice is that it prints, so it’s hidden now, but it prints a lot of, like, log messages. So, during the process, a few times I did like something silly, got the syntax wrong, and generally speaking, the error messages were pretty helpful to point me to where stuff went wrong to debug it and move on quickly.

Ben Sherman: It’s really not a bad habit, just having the make install, the make assemble, make install in there because it’s all cached. So even if you’re not changing it between runs, it’s going to be very fast. So it makes sense.

Rike Hanssen: Yeah, and it just avoided me from wanting to throw my laptop out because I felt like, “But I made the change. Why are you not picking it up?” And it’s just me not getting the command right. I would put it in a config if I could, but I didn’t figure it out. And then this was easy enough to do.

Yeah, and in terms of development, so we could, you get this scaffold that has a lot of files in, but in the end, I think the only files I truly looked at were in this folder called source. And then we have like a folder structure here. And then for this plugin, I didn’t touch everything you could possibly do. I think the main one I touched were the emoji observer file. Just to, when before the pipeline starts, I wanna do something. When the process completes, I wanna count stuff, and when the pipeline finishes, then I wanna print out something. So these were the main events I wanted to interact with and just look at and try to understand what they were doing.

Phil Ewels: Is a lot of emoji, 12 different themes and how many emoji per. Did you write all those out by hands? I can just imagine you there with the emoji picker.

Rike Hanssen: I really wanted to learn. So I started out just writing everything myself, no LLM, no Claude and then no Seqera AI. And then one, once I got the hang of it, I was like, “All right, now go ahead and add five more options or 10 more options.” So yeah, it’s been working. So this nice for me to learn really how things are working and where I can find something and to just understand it a bit better. And when I was a student, I started out with Java, but it’s been many years since I’ve written Java code or Groovy, Java-adjacent stuff. So it’s been really fun to get back to it, look at this stuff again. And then in this class, I think you get some of these functions as a skeleton already, so you can just start to fill them. Or if they’re not as a skeleton, then they’re definitely in the documentation to grab on various events that happen, like before the workflow initiates or on error.

Ben Sherman: It’s probably worth talking about this a little bit because the plugin template, when you use the nextflow plugin create command, it does add a lot of these sort of skeleton classes for you. And the Observer is one of those. I wanted to call it out because I think the plugin template by default has this observer that just says hello on workflow start and goodbye on workflow completion, or something like that.

So just, just make sure to delete the stuff that you don’t need. Because I noticed there were some people, like they would make a plugin and then they’d forget to delete the observer and then they would run their plugin and it was supposed to be something completely different. But then it was also saying hello, goodbye on the run. Of course, in your case you just replaced all of that with emojis. So it didn’t happen in this case, but to your point, yes. There’s, the observer here, there’s an interface for this in the Nextflow source code. We probably need to document it explicitly in a Nextflow docs so that people don’t have to, like, go digging through the source code.

But if you look up that interface in the Nextflow code base, it lists all the events and documents like when they happen. But yeah, it’s like you said, there’s an event for like, when the workflow starts, when a task is created, when it’s running, when it’s completed, when it fails, if the run fails, when the run completes. So you can really hook into all these different sorts of lifecycle events. And that’s really what makes this observer so powerful. That’s why I love this particular plugin. It just shows how you can use this observer for anything from Slack notifications to integrating with some third-party API, to generating a report, like nf-prov uses the observer to generate all of its provenance reports, or playing games with emojis and Numberwang. So you can really do all kinds of things just with this one observer.

Phil Ewels: And these are all overrides, annotations here. So presumably if you don’t need any of these hooks, you can also just delete this code or even delete the whole file, and you’re not going to break anything.

Ben Sherman: Exactly.

Rike Hanssen: I’m, I think this is like all these functions were documented in the Nextflow documentation. We are also, or Jon is also touching on them in his training. So, I just copied all of them, the ones in that I was interested in, on playing around with. Think the onFlowError, for example, like what happens when the workflow runs into an error, what do I want to happen?

In my case, I just wanna print an error emoji, which is, I don’t know, something like a red X or fire or something like that. But sure, you could do something slightly more sophisticated, like maybe sending a notification, “Hey, your workflow failed, you might wanna look at this,” for example. And then for the confetti, Phil, before, because you were asking, I installed this tool called CLI confetti, and then I’m just calling it and run it for one second.

Phil Ewels: Nice. I, previous podcast and stuff, I’ve did this Node-RED automation and that had a, one of the demos for Node-RED, I launched confetti when my own pipeline finished successfully. Confetti part took me longer than the whole rest of the automation combined, I think, ’cause I ended up web servers and all kinds of stuff to try and launch confetti on my screen. So I, I have a lot of respect.

Rike Hanssen: Yeah, I think I tried out five different tools or something until I found one that worked best for my use case. I took a, a bit of a distance from setting up my own web server. That sounded too scary, but I looked at the one that you did for Node-RED.

And then, yeah, there, there are basically all sorts of events you can hook into. Also process cached. I’m just counting things and incrementing progress VARs and incrementing emojis that are telling me about how many events were cached and so on. But I think I had the easiest time finding a use case for the Observer class here and, like, it’s also because I’m writing more Nextflow code than I’m, than dealing with executors in that sense. Again, it was just very obvious to figure out, oh, like when a process finishes, I want something to happen. I want something to go off. Did also try out some other things, but I couldn’t really, for this particular, for this emoji use case, I feel like everything that was related to process finishing or starting or workflow starting or ending was a bit more obvious. Maybe if I find time I can start adding in special emojis for executors that things are running on, or just make use of other knowledge I’m gaining about the workflow here.

Phil Ewels: Lots of cloud emojis or something.

Rike Hanssen: Yeah.

Ben Sherman: And I’ll say this, any day that you can write a plugin and get away with just using the observer, and that’s all you need, that’s a good day. That’s a very straightforward plugin. You could basically inject it into any pipeline, no additional configuration that, I mean, maybe a little bit config, but no changes, pipeline code. It just slots in and works. That’s the greatest kind of plugin in my view.

Rike Hanssen: Yeah. Now I just need to prevent this from being put into every nf-core production-level pipeline, because maybe not everybody wants populated with this.

Phil Ewels: I was going to just say a little side note is that remember that also we’re used to importing plugins into pipeline code, but you can also install plugins globally, right? So that they run for every single Nextflow pipeline you do. So if you import this into your .nextflow/config file in your home directory, then you would get this for every nf-core pipeline without waiting for the community to adopt confetti. So that’s good news for me.

Global installs & custom config scopes

Rike Hanssen: One of the most powerful things is that we can, if you want Slack notifications, you can set up nf-slack, if you want, if you don’t have Slack, but use Teams, then use the nf-teams plugin instead, and the pipeline author doesn’t need to worry about maintaining that code in the specific pipeline anymore.

Ben Sherman: I saw that had an example config file with some emoji config too, right?

Rike Hanssen: Yes. At the config itself or wherever this was implemented?

Ben Sherman: Yes, this config file here.

Rike Hanssen: Yeah.

Ben Sherman: I saw that. People who are watching, you’ll notice that the config settings are being highlighted, language server right now is yellow. That’s ’cause the Nextflow language server doesn’t recognize these config options. But this is also something that I wanted to call out as something that’s now possible with the plugin system. I don’t think it was really possible before, but you can now actually define your own custom config scopes. I mean, you could before, but now you can do it in a way even the language server can pick it up.

Now this is a bit more of an advanced concept, so, I’m not surprised that you didn’t get into this in your first iteration, but you could actually do this by declaring it, it’s called a config scope. You would create a class, a Groovy class. You would probably call it like emoji config or something. And you have to implement this interface. And then there’s a couple annotations you have to use. When you build your plugin, Nextflow will slurp up all those config settings that you define in your Groovy code, bundle it with your plugin binary that gets uploaded to the plugin registry, is like a JSON file attached to the actual code, and then the language server, when you specify the plugins and you say id 'nf-emoji@0.1.0', it’ll actually look up that plugin in the plugin registry, download this JSON file that tells it what all of the config options are, and then use that to validate your config. And so then that’s how you can get those yellow squigglies to go away. So just a pro tip for everybody out there.

Like descriptions and stuff in your config class, then it’ll even show like a hover hint of what the config setting does. So again, it’s not necessary to make the plugin work. It’s if you wanna make it work with all the language server goodies.

Phil Ewels: I spotted a couple of warning log messages when Rike ran Nextflow as well. And this will make those go away as well. Is that right?

Ben Sherman: Right. This is covered in the training as well, so you sort, you’ll have to do this.

Phil Ewels: Rike, is there anything left for you to show us in the plugin?

Rike Hanssen: Think that’s probably, those are the most important bits, I think. Yeah, install it so you get little horses racing around your pipeline when you run it. It’s a fun learning experience, as Jon said. We learned by playing. You can do the training when you gotta get out and draft your own plugin. So let me know if you find any additional things you’d like to add.

Phil Ewels: Awesome. And then, Jon, you were just mentioning that the stuff with the config scope is in your training. Tell us a little bit about how that came about and where people can find it and what kind of stuff is in your new training material?

The new plugin training material

Jon Manning: Okay, so yeah, this is the front page of the training. I’m not going to go over this in too much detail ’cause we’ve actually covered quite a bit of it today. But essentially you can go to our training portal at training.nextflow.io. And if you scroll down to Side Quests and into Plugin Development, you’ll find this page.

I should say that all of the software required to, to build and install plugins is actually already present in our pre-prepared learning environment. So you don’t need to know anything about how that sort of stuff is installed. So you can just go here and learn from the start, from the very ground up, how to build plugins.

It starts with just a discovery of kind of what plugins are out in the wild, or a couple of examples. So we actually talk a lot about the, after we intro there, the basic concepts you learn today about functions and workflow monitors and so on. And we talk about nf-hello, that example that Ben was talking about that was there before.

And then we move on to, for example, nf-schema, which is the nf-core community-wide, widely used plugin for parsing sample sheets. And then we also move on to describe how to use the CO2 footprint plugin, for example. And we talk again about how to install plugins and do some configuration as well, because this CO2 footprint plugin has some configuration associated with it.

And you can actually go and, and try those configuration options out. And then in, in those, in subsequent parts of this material, you go and actually learn how to do things yourself. So if you go to part two, you actually get to run that command that Ben was talking about, running nextflow plugin create.

And that will create that scaffold for you and you build from there. And there’s lots of helpful tooltips explaining what everything is as you go along. And then we move through subsequent parts where we add custom functions. And we add the trace observers as Rike was talking about in those last parts there.

And we also talk about something we, we haven’t talked about actually today, which is testing. Because one of the benefits of factoring out your functions into somewhere that’s shared is that you can actually have centralized testing. You can make sure in one place that those functions work as they should, and then you can be very confident about deploying them to your workflows as well.

And then importantly we finish off the material by talking about configuration. First of all, in, in simple terms, just adding in to configuration. And then if you scroll down in this part three is it was where we talk about this config scope thing that, that Ben was talking about, where you actually formalize the configuration option so that Nextflow is able to, or the Nextflow plugin in particular is available to interpret those things and be, and show you helpful input on, when you’re developing, for example.

Just go, if you wanna know how to develop plugins, as Rike has done so successfully, feel free to go here and look through and learn from the ground up. And we basically assume no knowledge. So if you’ve never developed an Nextflow plugin before, go here and it’ll teach you how to do it.

Phil Ewels: I think this looks so helpful and actually I need to sit down and do your training at some point. But the testing is a really good point. Again, I’m coming from a Python world myself, like I’m used to pytest and things like that. What framework do you use for Groovy, and is this Groovy test frameworks for Nextflow plugins, or is it something bespoke?

Jon Manning: There is a special framework, it’s called Spock in, in, in the Groovy world, and maybe Ben should illuminate a little bit more on this because I’m a bit sketchy on the detail.

Ben Sherman: I can talk about this a little bit because a Nextflow plugin scaffolding is very much modeled after the way that a Nextflow code base works. It’s Groovy code organized a similar way. We use Gradle to build the code, and we use Spock to do unit tests. Now depending on what kind of plugin you’re writing, unit tests may or may not be helpful. It just depends. Unit tests can get tricky sometimes, like if you’re having to mock out different APIs and things like that. But you can do that, especially for testing if you write custom functions. Unit tests are a great way to just say, “Hey, when I call this function with these inputs, they produce these outputs.” Very straightforward. And so Spock facilitates that. You just write Groovy code for your tests. And there’s some nice syntax sugar for writing tests.

And then what we also recommend is that you have some kind of end-to-end test, like a Nextflow pipeline in the plugin source code, even if it’s just the small, little pipeline, just so that you can run a full pipeline with the plugin included. Test all the different functionalities that the plugin adds in the same way that users will actually use it, just to make sure that everything fits together correctly.

Phil Ewels: And then, Rike, you mentioned that you used AI to come up with some emoji suggestions for you, but did you use AI any more extensively than that? And how did it cope with this kind of, this niche?

Rike Hanssen: In the beginning I really wanted to learn, so I wrote most of the stuff myself. Just because AI makes it really easy to skip over the hard part and in the beginning and let it scaffold everything. But later on, once I like out my way around the code base, knew where things were, got the feeling for how to proceed, I used various LLMs.

I don’t remember anymore which ones exactly, but it worked really well. So I told a little bit more, this Groovy code, pointed it to some of the docs and so on. But, it, it worked super well and also caught a lot of things that I wouldn’t have known about. But it was probably been very painful for me to be debug on, like, concurrent modification exceptions for variables that I was in various places at the same time. If I would’ve done this without, it would’ve probably taken me quite a while. And also a lot of test cases to really figure out what’s going on. And these AI tools are just really good at, and detecting it. And then for the more silly parts like emojis and so on, it just takes away a lot of the tedious work of, the space theme has taken for failed, it’s taken exploding or something, you just get a scaffold of it and then just fine-tune as you go.

Phil Ewels: One of the things I was…

Rike Hanssen: Impressed.

Phil Ewels: One of the things I was thinking about specifically was tests as well, ’cause I’ve written a few Spock tests myself as, for pull requests going into Nextflow, some of which were before the era of capable LLMs, and I dunno, I always found, like you say, Ben, there’s lots of syntax sugar, but I always find myself tripping up on them somehow. Now that Claude and Seqera AI and the others can write unit tests for Nextflow and for plugins, presumably must help.

Rike Hanssen: I think most of the tests I ended up writing in the end with LLMs. Also because then I figured, okay, I know what the output is, I just want you to set up the testing now exactly as things are right now, and then we can start iterating again on new features and just make sure the test pass. Think one thing I noticed during the plugin development, there’s a lot of reporting, like HTML files written on the, and their status of it, and you can just look at it and review them. Which I found really, for me, really convenient to just review and understand what’s going on, what is working and what is not.

Phil Ewels: Am I right in saying there’s also some GitHub Actions, automations and CI set up in the template, or have I made that up?

Ben Sherman: I believe there are some automations. I would have to refresh my, my memory on what exactly they’re for. I think there’s one for testing. Yeah. So it’ll, a CI for building the plugin and then testing it against that, that validation pipeline I was talking about. Because I think the template comes with a default validation pipeline. And so it will build it, run the unit tests, run the validation pipeline, and then you can customize that your needs.

Phil Ewels: Nice.

Rike Hanssen: Yeah, I recall correctly the whole framework and the make commands, they were fairly insistent also that I do write tests and not just skip over them for this being my own little, like, toy project. So I don’t quite recall what the trigger was, but I maybe when I tried to publish it or something or release it, I think there was a fairly strong bottleneck of like, you need to write tests before you can proceed here. Probably really good practice for all of us to just make sure things work.

Phil Ewels: Certainly doesn’t hurt. Jon, I, we said that the training material showed us is new. Has this been out in the wild yet? Rike is testing it internally, or how new are we talking?

Jon Manning: So we put the release of the materials out just today actually. So it, this is hot off the press. It’s been brewing in-house for quite a while and Rike and Ben actually and other colleagues have helped me refine it over time. It’s not just my work, it’s been a team effort and a product of some revision. So it’s been knocking around for a while internally, but it’s just a day out in the world.

Phil Ewels: Awesome. So guess people should check it out. And then where’s the best place to go? Is it community.seqera.io…

Jon Manning: training.nextflow.io. And you’ll find the material under Side Quests on the left-hand side.

Phil Ewels: Yeah, I meant feedback, if people, or have any questions, they can go to community.seqera.io.

Jon Manning: Ah, yes. That will work.

Phil Ewels: Maybe just to, to wrap up and the final part, you talked a little bit about publishing your plugin, Rike, and we touched on claiming a namespace earlier and stuff. But what does that final part of the picture look like? How does that final approval and publishing step work?

Publishing and approving plugins

Rike Hanssen: I’m going to share my screen just for people watching. So we have this website registry.nextflow.io that you can log into. I’m already logged in now, you can just go ahead and claim your plugin. So I can see there are various tabs here. One of them is My Plugins. I can go ahead and request the plugin ownership, fill out various things like the name, the provider, where it lives, a description, why do we definitely need an nf-emoji plugin and why it’s really important. Then you can submit a request, when it was approved pretty quickly. I wanna say maybe same day or day after.

And then when I wanted to publish my plugin, I could generate an access token and I wanna say plug it into the GitHub Action or locally, bit fuzzy on the details, but I got an access token and then it just now keeps publishing my plugin whenever I do a new release. It was very straightforward. The longest part was waiting for someone to approve it, which was the same day, so not exactly long, and now I don’t have to wait anymore. It was the one-time, one-time thing and now I’m independent and can just go ahead and keep publishing it.

And then here you can also see on the All Plugins side, which are the plugins have already been claimed. You can make sure you’re not duplicating names and just start off with an, with a unique name. Just to begin with.

Phil Ewels: Nice. And then there are a few different fields in that claim. I mean, Ben, when you get a request coming in on your side, what kind of things do you think about in terms of what to approve and not?

Ben Sherman: We don’t really have a super formal process for this. It’s mostly, does the name, is the name appropriate for what they’re doing? Is it specific enough? Like, there was someone in the community who a Python integration. They called it nf-python. We definitely scrutinize that one a bit more because that’s going to be a very broad name, nf-python.

We wanna make sure that the functionality actually justifies that, and it did. Aside from that, we like to see the plugin source code. We like to see that it be open source, just to benefit the community. We like people to give a basic description. Beyond that, that, that’s about it. As long as it’s just, and most plugins that we get are reasonable. Even the emoji plugin, that’s pretty reasonable in our book. It’s, it makes, like, we understand what it’s doing. So we mostly, we usually just approve it. It’s usually not a big deal. And if there is an issue, then we’ll probably reach out to you, Paolo or I, or you, Phil, we would reach out to the person’s email and discuss whatever issues we might have. But I don’t think that kind of thing has come up yet.

Phil Ewels: And you said that you need to a source code. And I see on this page as well, it says something about including a license notice. So I guess open source, saying it’s a requirement for plugins on the community registry that they’re open source?

Ben Sherman: I think yes. I don’t know if we’ve formalized that anywhere, but that is our basic expectation, that it’s open source.

Phil Ewels: And then this is a bit of a difficult topic, but what about people who don’t wanna publish their as open source or they don’t want their plugins on the community registry, but it’s more for inner source collaboration or working in-house with their plugins. Like, how should people go about distributing their plugins internally?

Ben Sherman: It’s certainly a legitimate use case. I think it’s possible now, but it’s a bit hacky. I think you have to continue to use the old JSON index approach where you basically have an internal JSON file that just lists all of your plugins that you’re using. Long term, what we’d like to do is, and in fact this may already be possible now, I think if you go to registry.nextflow.io you can actually look at the OpenAPI spec for the registry, basically like the API that Nextflow uses to download plugins. Yeah, it’s registry.nextflow.io/openapi. You can download the OpenAPI spec. If you can implement your own registry backend and as long as it provides that API, at the very least the API for downloading plugins, then you could just use that internally. That should just work out of the box because Nextflow is not tied to the community registry. I think that’s like a config setting that you could point Nextflow to your own internal registry. This is also something that we’re looking at internally, this could be something that we provide, even as a managed service, if customers are interested.

You can definitely reach out to us if you’re looking into that. But I think at this point this would be the approach is to basically build your own little sort of mini registry and go from there.

Phil Ewels: Yep. Watch this space for new future developments. Hopefully we can make, provide something a bit more user-friendly to folks. Any closing thoughts from anyone? Is there anything we’ve missed on the current state of Nextflow plugins?

Closing thoughts

Ben Sherman: A lot of us iterated on Jon’s training, I cannot recommend it enough. It is a great walkthrough of plugin development. Chris’s docs on the Nextflow website are pretty good. But the training is just really good at spelling everything out, walking you through a concrete example. If you want to get into plugin development, I would start there.

Phil Ewels: No disrespect for Chris. His docs are also excellent.

Ben Sherman: They’re complementary, they go together.

Rike Hanssen: Yeah, I think that’s the, my biggest takeaway from the last couple of weeks is that plugin development isn’t as hard as I thought it was going to be. And with all these new resources available it’s been a lot of fun to, to play around and learn it.

Phil Ewels: I feel the odd one out on this podcast is the only person who hasn’t done the plugin training. That’s my, my evening sorted. Great stuff. Well, thank you very much for joining me, all three of you. It’s been a pleasure as always, and thank you everyone who’s listened to the podcast.

I hope this has been useful. Like I say, the previous plugins podcast episode was super popular and hopefully we’ll see a surge of applications and claims for new plugin names in, in the next week or two. As all of you’re equally convinced that the, from Jon’s training, that plugin development isn’t actually all that scary. And you, all these ideas you’ve had fermenting in the back of your head for the past years can now come to the fore. Thanks very much for listening to the podcast and we’ll see you in a future episode. Thanks all.