Share your experiences selling Dagger into your organization

[blazing swift]

This post is hopefully a place we can collect our shared experiences and tips for selling Dagger into an organization

[oblique crystal]

Love this. cc @full saffron @wheat zinc @brittle yoke

[blazing swift]

I could write a blog series on this lol

[topaz cosmos]

This would be really useful for me. I've had no issues selling it to developers, but I would need to focus more on the positive business impact to pitch it to executives one day ❤️

[full saffron]

@quiet raptor @tulip pawn , I think you'd have some thoughts here too 😂

Yes, I love all of this. We've seen this request many times, and I was just talking to @brittle yoke about how we need content around this.

Start throwing things that you need from us here, and we're on it!

[blazing swift]

I'm currently working on a presentation to sell a new vision for developer experience at a client of mine, so I can only share so much right now... because deadlines...

tl;dr

• had a specific and narrow problem where I was able to pitch and get approved a limited use of Dagger, this went well
• got approval for a larger PoC from the CTO, this was in part supported by a longer running conversation we were having around developer experience pains and possibilities, this still lives on a feature branch, but the Dagger portion looks to have technical improvements
• Myself and the CTO had a meeting with the Tech Lead of the platform, he gave a lot of pushback, particularly around framework and abstraction burnout
• We reflected and decided we want to pitch the changes differently, focusing more on the pains and problems, leaving implementation details aside, but bringing in CUE + GoCLI + Dagger near the end to paint a vision
• I will update more next week. I have a review of the slide deck tomorrow with the CTO and we have a meeting monday with the Eng PM / manager

Some of the points we want to highlight about Dagger are

• We are proposing to (finally) containerize the full CI process, we want it to work the same locally / remotely. There is an aspect of showing alternatives and that we retain some of the pain points we want to remove, so Dagger becomes the natural choice
• We have these intermediate images that are a PITA (think a shared runtime alpine image with some packages that all the go services FROM and copy a binary into), currently this is tagged and involves extra work & CI. Dagger means we can "remove" this intermediate image
• We are seeing wins from the caching on a single branch that builds in isolation. We use Jenkins (shared build host) and expect more time savings across branches
• The terminal output and cloud UI have already received positive feedback from the developers based on the initial PoC
• We are proposing to remove Makefiles and Dockerfiles, by wrapping the Dagger SDK in a Go CLI (we don't use Dagger modules/functions)
• I've mentioned that Dagger is built by the same people as Docker, as their second take on how to solve this problem. This CTO doesn't think people like to hear that, but I have had success elsewhere using this as a sort of "social proof"

[full saffron]

Thank you for the summary. This is very helpful. Looks like your POC is going well, which is a win!

I have a few general resources that MAY help on the messaging side.

[blazing swift]

sure, this presentation is 80% screenshots of existing code from Bitbucket 🤦‍♂️

I am reaching back to some old messaging @flint minnow used to use around Dagger being a replacement for Make, like a glue that holds your CI together (because we have a lot of Make and a lot of pain therein)

[full saffron]

This might help with the general Dagger sell. It's our overview deck. Anyone can use it - https://docs.google.com/presentation/d/1VUPTvajY-RXlByUA9hyybDve5slVvqGCGDBbwpfciM4/edit?usp=sharing

Case studies usually help to share too - https://dagger.io/case-studies

[blazing swift]

I think I've seen that before, I'm definitely hitting almost all of what's in there, but making it very specific to the client code base and dephasizing the Dagger details

We honestly don't want the devs to even be aware of Dagger or other implementation details.

[full saffron]

Yeah, that is great. If you'd like, we'd be happy walk through what you have or join your team calls to support.

[blazing swift]

Their primary interface is a config file and a CLI, though they see the nicer build logs and can share links to the UI for local build issues

I don't think most of the dev team will have a strong opinion on Dagger yes/no. It's mostly about getting buyin that there are these problems and we should allocate time & resources to addressing them.

I've been working on selling the CTO on tech debt sprints for a long time (years). I think his approval to create the proposal in the first place is that effort paying off, coupled with the tangible outcomes from experiencing the pains of the debt more frequently

[full saffron]

That is exactly what @tall lion did 🙂 https://dagger.io/blog/daggerizing-production-pipelines-how-openmeter-overcame-push-and-pray

[blazing swift]

The CTO is pretty on board with Dagger, he did give you his credit card already :]

It's that old sales situation where you have internal champions and detractors

the champion is pushing to reframe the proposal to win over the detractors

We are intentionally have a meeting with the PM only, to build a coalition and test out our new angle

[full saffron]

We are actively looking to build material to support your efforts. If is helpful, we are happy to join your calls or review any of the content too.

Thank you for starting this thread because we have many people in the exact same situation as you.

Your experience was the catalyst for this page - https://docs.dagger.io/adopting

[blazing swift]

I have to sell them on expanding our use of CUE and Dagger at the same time, while mentioning them by name as little as possible, because of there is generally a framework burnout cycle going on in the dev world

I recently saw a new argument around not adopting new frameworks or even versions of stuff you already use... because the AIs won't know about it yet, so wait until the AIs can be retrained and regurgitate

[full saffron]

@steel bear @stoic sleet might have some tips here too.

[full saffron]

yeah, this one can be hard depending on who is in leadership.

If there are thought leadership pieces that you'd want to see from Dagger, let us know. Personally, I've seen those type of assets help convince the C-suite.

[blazing swift]

selling to the C-suite is usually about more profits or lower costs, preferably the first. With developer tools, you are often pitching cost savings or increased productivity. (or another way, removing pains rather than opening opportunities)

[tall lion]

Couple thoughts:

Selling a solution (to both tech people and business) boils down to pain (or some other feeling). Selling a solution that changes a status quo will naturally have opposition, because it's a risk, an unknown variable (you can't guarantee success). It's not even about Dagger. Most arguments I've seen against any technology can be dismantled fairly easily if you understand what's behind it.

The hard truth is, you need to sell the pain. Why it's going to be unbearable if the situation doesn't change. Project a future so dark, that they run screaming to you for a solution. (Is this a joke? Maybe...maybe not)

The easiest way to do that is metrics: how many hours do developers waste on their pain (eg. shitty scripts that never work, push'n'pray, etc). Maybe even a day? How much frustration does that cause? (Like literally, how many arguments do people have each week due to the shitty environment they have to work in?)

Now turn those metrics into time and money: if every developer wastes a day with those scripts every week, then 40 people waste 40 man days a week, that's more than 2000 man days a year. How much money are you bleeding into the void? How much would it take to build a PoC that changes the situation? How much money would you spare if you improve only 10%?

Now, about some of the points raised above:

Hiding Dagger...is probably somewhat counterproductive, because the point is settling on a solution that people don't try to avoid touching. One of the reasons why traditional CI s*cks today, is because noone wants to touch that 1000 LOC bash script. Maybe noone knows how it works anymore.

Dagger provides a more familiar interface (programming language that somewhat mimics a Dockerfile) that developers can touch AND test easily.

Initially, it may be a good approach to prove it works.

Choosing the hook: in my experience, Dagger really shines in e2e test scenarios. Caching makes things soooo much better and the encapsulation of the test environment (or lack thereof) is usually what makes every other alternative fall behind compared to Dagger.

You could hide Dagger behind a Makefile in the beginning. If you do your job well, people will experience a decrease in pain and when they ask how you did it or look under the hood, they will find Dagger. At this point, you are in a much better situation to finish the sale compared to some early conversations where people's PTSD gets triggered about new tech.

Specifically about CI/CD: I think the biggest problem is getting people to realize how crappy the situation is today. Most people simply accepted that CI/CD is a pain. That when you push a change, it's normal to go for a walk or a coffee, because it will take 40 minutes for CI to build (which may be healthy for you, but isn't very productive).

I don't know (yet) how to make people wake up from the Matrix.

So if you already have some pain to work with, you are already in a much better position.

[tall lion]

A couple other metrics you can improve with Dagger:

• Reduced onboarding friction (if you are looking to hire new people, fluctuation is high or people regularly have to move between projects)
• Increased productivity can be expanded on a number of ways: reduced upfront costs for new projects (mostly time), etc
• Breaking down the silos: platform engineering and devs can work much closer to each other which has obvious benefits (again, depends on your organization structure)

[tall lion]

But....but...but...if noone adopts the new stuff, how will the AI "know" it should recommend something different (since noone uses it)? Even if AIs are retrained, if it starts giving "bad" advice first, wouldn't that be further deterrent from adopting new stuff?

This argument is bleeding from several wounds IMO.

[blazing swift]

One of the big things in selling, or even just presenting, is knowing your audience, making it specific and personal. Generic tips, suggestions from others, or detailed experiences are all good and helpful, but you have to convince certain people.

For my situation, the tools we are replacing (make, docker) are already hidden behind a single script the devs use for their day-to-day. So for us, focussing on a single technology is not going to help. Our solution is a composition of multiple technologies we have already running (CUE, Dagger), but are seeking to expand. I definitely don't want the devs running the dagger CLI by hand.

[tall lion]

That's most certainly true, although - and I'm admittedly not a sales person - most sales people (and maybe even professionnal interrogators) would agree that the challenge is in finding the trigger point for people that reminds them of the pain you are trying to address. The process is mostly the same once you found it: just keep pushing it.

Again, this is a bit dark, I know. 😄

[blazing swift]

The approach is, look at all these pains we deal with. Here's is the vision for how this works. Here are some of the ways we have already addressed this with CUE | Dagger, we propose to expand on them

[tall lion]

Makes a lot of sense to me. And at this point generic advice may really not be useful to you as you are working through the nuances of your organization and business.

[blazing swift]

My presentation is going to be like 40 slides of code screen shots (pain, in rapid fire) and a few solution. It's what the CTO wants to do to the devs

[tall lion]

I agree that the focus shouldn't be on the solution or the technology. The focus should be on the pain, the vision for a solution as you put it and how Dagger (or anything) fits into that picture.

Whether devs have to run dagger by hand or not...I can't tell.

Completely hiding Dagger behind custom abstractions may not be productive though. Maintaining those abstractions can be costly and as I said above, you want people closer to the solution, you want them to use, not stay away. And those abstractions are often part of the problem (I'm talking about vendor specific YAML files).

But if that's what works for you, I'm not gonna tell you not to do it. 🙂

[blazing swift]

We've definitely moved towards creating something like BuildPacks for the devs. They rarely even look at the makefiles or dockerfiles today. Just had one where they needed to add tzdata to the runtime base images and they didn't even realize that one builds on the other and they only needed to apk add in just the first one

[tall lion]

That is until everything works. One of the areas where Dagger shines is you can look at and fix potential issues fairly easily as a developer as well.

[blazing swift]

I have my reservations about the devs not understanding how their code is built.

[tall lion]

But the buildpacks for devs angle is definitely something that can work very well with Dagger.

[blazing swift]

I don't think this is necessarily true. Yea they might be familiar with the language, and recognize some Docker stuff, but there are definitely concepts that are new and they would have to learn, i.e. they have to become familiar with a new framework

[tall lion]

That is true, but isn't necessarily a bad thing. I mean, you have to keep learning as a developer every day. If you have the opportunity to learn AND fix your problems without waiting for others to fix them for you...I think that's the dream for developers: not getting blocked by a waiting for platform eng team sitting two floors above them to reply.

Anyway, I think this isn't really about the selling part anymore. How you integrate a new tool is totally up to you and your org. Do what works for you.

[blazing swift]

In my experience, most devs want to focus on the code they are responsible for, the product and business logic, rather than all the stuff around it. It's why tools like next and hosts like vercel have become popular. They'd rather be building something cool than doing ops work all day

There are a number of startups that have been pushing the idea that "devs can do their own ops with our tool", but it's not their expertise and they tend to do the minimal to get what the think works. You end up with a different solution from each team, and then eventually an expert has to come in and clean it up, create consistency, sanctioned paths, and guard rails

but there are a ton of companies and just as many situations and approaches, ymmv as the saying goes

[tall lion]

Well, with Dagger, you can go both ways: you can build those sanctioned paths and reuse them across projects or you can empower developers to own their "code" end-to-end.

[tall lion]

I think tools like vercel have become popular mostly due to the "it just works" mentality...until it doesn't. But it certainly just works for a large number of cases.

I don't disagree that devs often dread the ops work...but part of the reason is the "framework": the experience is bad.

[tall lion]

IMO this is an organization and talent question which is specific to every company, so there are no right or wrong answers here.

Seasoned people probably like having their own way, don't want to wait for others. Less experienced teams may need more guidance.

There is also a compliance angle to consider, especially in larger organizations.

So there are lots of ways to approach the implementation part.

[tall lion]

Would be nice to put some numbers next to your story to get a clearer picture of what you are dealing with. Like how many developers/teams/projects, how long did you run the PoC, what metrics you monitored and how those changed, etc. If you can share any of those, of course.

Could even be a nice case study for Dagger. 🙂

[blazing swift]

We have less metric-y stuff and more things like, "in our monorepo, we have developed hidden dependencies. Here are three examples, this is... not good... here's how we think they came about...
(1) we catch this with Dagger (2) we need a moethod & place for the legit use-cases (3) we need to do X, Y to untangle the others

[mortal cairn]

Completely hiding Dagger behind custom abstractions may not be productive though. Maintaining those abstractions can be costly and as I said above, you want people closer to the solution, you want them to use, not stay away. And those abstractions are often part of the problem (I'm talking about vendor specific YAML files).

This is certainly something I struggle with at my org. When presented with dagger (to platform/devops devs) the immediate instinct is "how can we hide this from the developer". I can relate to both your points around this. There's a tendency to think that an abstraction will serve a majority of developers better than giving them control and ownership of their ci/cd tools. I am starting to think a good way to approach this is a bottom up strategy where you expose and train developers in dagger, create useful everyday tools they can use on their local machines and then get those developers to ask for "more dagger". Then we can be like "hey look! you can run your entire CI on your local machines using the same tool. No more push and pray".

[tall lion]

A top-to-bottom approach can certainly work with Dagger as well. For example, if you provide entire pipelines to developers, you can always break those down to lower level building blocks as you progress.

You still get a lot of benefits that way.

I probably wouldn’t go as far as completely hiding Dagger behind an abstraction, but a simple Dagger call to a remote module can certainly work in the beginning.

[modest mantle]

A little unstructured context from our journey so far...
We've rolled out dagger to our enterprise (+500 devs) and we're 2 months away to be "in production" with the new Dagger based CI platform. We developed modules that provide the common functions CI does nowadays for our developers (building / testing / pushing artifacts / deploying to our systems).

• Understand the pain points of your devs, that is paramount. I agree with @tall lion , if you get the emotinal trigger of frustration from your devs, and you address that. Your adoption is going to be really smooth. In our case (and I think this is relatable to many CI systems today...).
  CI is difficult to debug locally. A failure in remote involves HOURS of push & pray to understand the differences from local execution to remote execution
  Feedback loop: Push & Pray / Push and go for a coffee. The total waste of time and break in developer focus is real with the current CI state nowadays as soon as your build does anything complex for more than 10m
  Drift between what you test locally and what you test in CI.
  Jenkins + groovy (the current/previous setup) is not seen as a career progression. Devs won't like to spend time in a technology/language that don't help their CVs/career development (Go / Python / Typescript are much better seen)
  Reusability of each developer's CI code is low between different teams
  Dagger adresses by design many of these pain points, so it was easy to fit it into the solution

If you already have a working system, although painful, don't expect developers to adopt a new system without some resistance. Their main focus is not getting CI faster or more reliable, but to deliver business value in most cases. Ci is a by-product that, until they need to modify, they won't try to even look at it (this is our experience anyway). Take the advantage of that situation: WHEN they need to implement anything new, that's when you can introduce the new solution. Once they try how better the experience is with Dagger, they will go back and refactor the rest.
This could be a small side-project they are working on, a new functionality that can be completely separated from their current CI.
We've done a full replacement. We've built a completely new platform from scratch using Dagger, but developers don't need to adopt the whole shebang straight forward. They can start experimenting with smaller features.
This helps also with adoption

At the beginning of our journey, we started by abstracting dagger, through a makefile. (everyone at $company uses make).
This didn't work for us. As soon as your engineers are trying to customise something, or are curious enough, they'll try to look under the hood and your abstraction falls apart. Developers are keen to code. If you give them a familiar language, they'll be confortable looking/creating/improving their pipelines

EVERY SINGLE TIME an engineer moved from their Jenkinsfile to use dagger and the modules we provided for them, the feedback has been amazing. They go back to the rest of their code repositories and start adopting it more and more.

[tall lion]

Congrats!

[modest mantle]

TLDR; The Selling Point
Developers: Identify your company current CI problems, find that "painful" process that developers suffer now. They will be onboarded with the solution faster than you think 🙂
Exec: Identify how those pain points affect the metrics that they care for. How many hours are wasted in CI is a direct translation to $$$ of engineering hours.

We'll collate all of our journey soon ™️ and prepare some slides to share with the community, with some real metrics on how the world has improved in our $company 🙂

[blazing swift]

Perhaps my problem was that I have already introduced Dagger concepts without using Dagger

[tulip pawn]

haha, It wasn't that hard of a sell. I was asked to look into it initally. The harder sell was when talking to DEVS.

[blazing swift]

So another experience I had very recently was...

1. Was on a call with my main client, to help with deciding which security vendor they are going to buy
2. We all introduced ourselves, and when I was done, the CTO told Aqua I was also an "Unpaid Dagger Advocate"
3. I was later on a debug / support call with on of Aqua's engineers and showed him how were using Dagger with their tool in our monorepo
4. The support engineer became a Dagger fan instantly, repeatedly said he thought Dagger could really help with their upcoming effort to add Monorepo support to their products
5. I kept telling him that adding labels would probably be enough, but he was quite adamant about using Dagger
   (I'll reach out to him and see if he wants an intro to the Dagger team, I can only imagine the answer being yes)

[quiet raptor]

I was giving this some thought, I'm going to be careful about what I share because they clearly take privacy and security extremely seriously.

Blackstone has an entire process when it comes to adopting a new tool, you have to write documentation justifying adoption.

From what I sense, it's easier getting things moving if there is a quantifiable value that can be communicated be deemed as high priority enough over pre-existing tasks. Initially when I spoke about Dagger, my team understood the value, specifically in terms of testing consistency, but at that point it wasn't deemed as high priority.

However, just recently our VP mentioned that he wants me to start building out more pipelines for other apps we have in our pod. The pipeline I built for our main app is going to be used as a template for other apps we have. I have spoken to him about inconsistent testing and how this slows things down at least for me (ie. I write integration tests using something like SQLite locally, while in prod we run our tests against a different DB which is a lot stricter around things like foreign key constraints for example).

I already know this and can write more robust tests, but we plan to grow the team and have other contractors and devs working across different apps.

We are all in agreement about setting up the infrastructure to run tests locally in a containerized environment simulating our pipeline. This will be a priority over our next sprint (February).

The VP and I have one on ones weekly. My secret plan has been nudging conversations towards this over the weeks. Buy-in from our VP wouldn't really guarantee anything given the internal process of adopting a new tool, but it's a lot better than just me proposing it in isolation

[full saffron]

Thank you so much for sharing @quiet raptor !

If there is anything in terms of material or even additional onboarding tools that you'd think would be helpful, let us know. I'll be happy to add it to our backlog to work on.

[oblique crystal]

@kind jungle would you mind moving this message somewhere else please? We understand it's important for you but it's not directly related to the topic of this thread and it's not adding valuable information. It'd be great to hear your story about selling Dagger into your org and discuss about feature requests in a different forum please

[kind jungle]

Removed. Thanks

[forest nacelle]

I didn't really have to sell Dagger as we're a small startup, so when I was brought on I had a pretty wide berth re tech choices. With that said dagger has helped in this situation immensely with how much easier it is to get complex pipeline functionality and how easy it is to create functionality that's useful locally and in CI/CD. We use dagger for all building, publishing, and updating our builds, but we also use it for creating local environments, local debug tasks, etc. This second part I didn't know when I chose dagger would be so useful, but its saved me a ton of time and kept our automated actions consistent from local environments all the way to prod.

[flint minnow]

That's great to hear! I know a few conference & meetup organizers who would love to see you give a talk about your experience 🙂 cc @full saffron

[wheat harness]

Hi! I'm exploring the feasibility of using dagger to solve the execution of thousands of containers to run test automation at scale within the constraints imposed by legacy systems we have here. We operate within a heavily regulated industry and with that in mind, is there a commercial support offering for running dagger engine in our own kubernetes (EKS) cluster? Many thanks!