Podcasts Technology Screaming in the Cloud CloudDev for Retail Companies with John Mille

CloudDev for Retail Companies with John Mille

Published: April 27, 2023, 10 a.m.

b'

John Mille, Principal Cloud Engineer at Sainsbury\'s UK joins Corey on Screaming in the Cloud to discuss how retail companies are using cloud services. John describes the lessons he\\u2019s learned since joining the Sainsbury\\u2019s UK team, including why it\\u2019s important to share knowledge across your team if you don\\u2019t want to be on call 24/7,\\xa0 as well as why he doesn\\u2019t subscribe to the idea that every developer needs access to production. Corey and John also discuss an open-source project John created called ECS Compose-X.


About John

John is an AWS Community Builder (devtools), Open Source enthusiast, SysAdmin born in the cloud, and has worked with AWS since his very first job. He enjoys writing code and creating projects. John likes to focus on automation & architecture that delivers business value, and has been dabbling with data & the wonderful world of Kafka for the past 3 years.


Links Referenced:



Transcript

Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.

Corey: It\\u2019s easy to **BEEP** up on AWS. Especially when you\\u2019re managing your cloud environment on your own!
Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need.

Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon\'s reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That\\u2019s S-Y-M-O-P-S.com/corey


Corey: Welcome to Screaming in the Cloud. I\\u2019m Corey Quinn. Today my guest is a long-time listener, first-time caller. John Mille is a Principal Cloud Engineer at Sainsbury\\u2019s, which is UK-speak for \\u2018grocery store.\\u2019 John, thank you for joining me.


John: Hi, Corey. Thanks for having me.


Corey: So, I have to begin with, I guess, the big question that I used to run into people in San Francisco with all the time. They would work at Walmart Labs and they would mention in conversation that they work at Walmart, and people who weren\\u2019t aware that there was a labs out here figured they were a greeter at the grocery store. Do you ever wind up with people making that sort of fundamental assumption around the fact, oh, you work at Sainsbury\\u2019s as a checker or whatnot?


John: No. But it actually is one of the\\u2014if you look at one of the job descriptions from Sainsbury\\u2019s, the first thing is, why would you join a retail company to do tech? And as it turns out, tech\\u2014I mean, I think retail companies, as any other companies in the world, rely on Cloud more and more and more. And I think that one of the things that is interesting today is, if you look at the landscape of retailers, I\\u2019ve heard many times people saying, \\u201cWe don\\u2019t want to go for AWS because we\\u2019re giving money to the competition.\\u201d And actually, I think AWS does a fantastic job overall giving you all the tools to actually beat them as your competition. And as it turns out, we\\u2019ve had really, really great success running a lot of our workloads on AWS for many, many years now.


Corey: On some level, if you can\\u2019t come to terms with the idea of Amazon as competition, you shouldn\\u2019t be using AWS, regardless of what industry you\\u2019re in, because their entire company strategy is yes. It\\u2019s very hard to start to even come up with industries that they don\\u2019t have some form of presence within. On some level, that\\u2019s a problem. In fact a lot of levels, that\\u2019s something of a problem.


Everyone tends to wind up viewing the world in a bunch of different ways. I like to divide companies into two groups. More or less it\\u2019s, is the AWS bill one of the top three line items at the company? And if the answer\\u2019s no, on some level, you know, that usually is an indicator that there\\u2019s a sustainable business there that, you know, both our grandparents and our grandchildren will be able to recognize, in the fullness of time. You absolutely have a business that winds up falling into that category, whereas, \\u201cOh yeah, I fix the AWS bill,\\u201d yeah, my parents would have no idea what I do and my kids don\\u2019t have much of a better one. It feels like it\\u2019s very point-in-time type of problem. At least I hope.


Technology is not the core of what grocery stores tend to do, but I also don\\u2019t get the sense that what you\\u2019re doing is sitting there doing the back office corporate IT style of work, either. How do you use technology in the overall context of the business?


John: Well, so we use it in a very wide variety of sense. So, you obviously have everything that has to do with online shopping, orders and all of those sort of things, which obviously, especially with the drive of Covid and being everybody from home, has been a huge driver to improve our ability to deliver to customers. But certainly, I think that Sainsbury\\u2019s sees AWS as a key partner to be able to go and say we want to deliver more value. And so, there\\u2019s been a number of transformation over the years to\\u2014and one of the reasons I was hired is actually to be part of one of those transformation, where we\\u2019re going to take existing infrastructure servers that literally\\u2014I usually say to people, \\u201cOh, are we doing an upgrade this month? Has somebody gotten their little brush to go and brush onto the hard drives to make sure that nothing is going to die?\\u201d And actually do that transformation and move over to the cloud in order to never have to really worry about whether or not they have to manage hardware and infrastructure.


Corey: It\\u2019s strange in that I never got very deep into containers until I was no longer hands-on hardware, managing things. I was more or less doing advisory work and then messing around with them. And you\\u2019d think given my proclivities historically, of being very unlucky when it comes to data, you would think that this would be great because, oh yeah, you blow away an ephemeral container? Well, that\\u2019s kind of the point. We\\u2019ll all laugh and it\\u2019ll re-instantiate itself and life goes on.


But no. Making fun of them was more or less how I tended to do approach them for the longest time until I started to see them a little bit\\u2026 well I guess less as a culture, less as a religion, and more as an incredibly versatile packaging format, which is probably going to annoy the people I know who are the packaging [unintelligible 00:04:58] for Linux distributions. How do you tend to view them? And how did you start using them?


John: Right. So, that\\u2019s a great question. So historically, I was a student at, I think the school were one of the original creators of Docker were. And one of the things that you learn when you do development at the school is that, you know, containers [unintelligible 00:05:18] new invention. Docker, I think, came on the platform as the way to, you know, give everybody a great framework, a great API, to drive the deployment of containers in the world and bundle them and ship them around the world, on your laptop and somebody else\\u2019s, and help a little bit with, you know, solving the problem of it works on my laptop, but not just on the laptop properly. Maybe.


It\\u2019s obviously gone viral over the years and I really enjoy containers; I quite like containers. What I find interesting is what people are going to do with. And I think that over the last few years, we\\u2019ve seen a number of technologies such as Kubernetes and others come into the scene and say\\u2014and trying to solve people\\u2019s problem, but everybody seems to be doing, sort of, things on their own way. And historically, I started off using ECS, when it was terrible and you didn\\u2019t have security groups per containers and all of this. But over the years, you know, you learn, and AWS has improved the service quite significantly with more and more features.


And I think we are today in the place where there\\u2019s this landscape, I think, where a lot of workloads are going to be extremely ephemeral and you can go [unintelligible 00:06:28], you know, wherever you want and you have a bit\\u2014if you have a platform or workflow that you need to have working in different places, maybe Kubernetes could be an easy way to have a different sort of sets of features that allows you to move around in maybe an easier way. But that also comes with a set of drawbacks. Again, I look at using EKS, for example, and I see okay, I have to manage IAM in our back now, whereas if I used something like ECS, for the whatever the [unintelligible 00:06:56] cloud vendor of choice, I don\\u2019t have to deal with any of this. So, I think it\\u2019s finding the fine balance between how you do orchestration of containers now and what works for you and is any sustainable over the time, more than about are you going to use containers? Because the chances are, somebody is using containers.


Corey: My experiences and workflows and constraints are radically different than that of other folks because for a lot of the things I\\u2019m building, these are accounts that are I\\u2019m the only person that has access to them. It is me. So, the idea of fine-grained permissions for users from an ARBAC perspective doesn\\u2019t really factor into it. Yes, yes, in theory, I should have a lot of the systems themselves with incidents roles being managed in safe and secure ways, but in many cases, the AWS account boundary is sufficient for that, depending on what it is we\\u2019re talking about. But that changes when you start having a small team of people working with you and having to collaborate on these things.


And we do a little bit of that with some of our consulting stuff that isn\\u2019t just the shitpost stuff I build for fun. But there\\u2019s multiple levels beyond that. You are clearly in a full-blown enterprise at this point where there are a bunch of different teams working on different things, all ideally going in the same direction. And it\\u2019s easy to get stuck in the weeds of having to either go through central IT for these things, which gives rise to shadow IT every time you find a corporate credit card in the wild, or it winds up being everyone can do what they want, but then there\\u2019s no consensus, there\\u2019s no control, there\\u2019s no architectural similarity. And I\\u2019m not sure which path is worse in some respects. How do you land on it?


John: Right. So, what I\\u2019ve seen done in companies that works very well\\u2014and again, to the credit of my current company\\u2014is one of the things they\\u2019ve done really well is build a hub of people who are going to manage solely everything that has to do with accounts access, right? So, the control, IAM, Security Hub, all of those sorts of things, for you. There\\u2019s things that are mandatory that you can\\u2019t deal without, you have permissions boundary, that\\u2019s it, you have to use those things, end of story. But beyond that point, once you have access to your accounts, you\\u2019ve been given all of the access that is necessary for you to deliver application and deploy them all the way up to production without asking permission for anybody else apart from your delivery managers, potentially.


And I think from there, because there is the room to do all of this, one of the things that we\\u2019ve done within my business unit is that we\\u2019ve put in place a framework that enables developers\\u2014and when I say that it really is a question of allowing them to do everything they have to do, focus on the code, and I know it\\u2019s a little catchy [unintelligible 00:09:33] a phrase that you hear these days, but the developers really are the customers that we have. And all that we do is to try to make sure that they have a framework in place that allows them to do what they need and deploy the applications in a secure fashion. And the only way to do that for us was to build the tools for them that allows them to do all of that. And I honestly haven\\u2019t checked a single service IAM policies in a very are longtime because I know that by providing the tools to developers, they don\\u2019t have this [will 00:10:05] to go and mess with the permissions because their application suddenly doesn\\u2019t have the permissions. They just know that with the automation we\\u2019ve providing them, the application gets the access it needs and no more.


Corey: On some level, it feels like there\\u2019s a story around graduated development approach where in a dev environment you can do basically whatever you want with a big asterisk next to it. That\\u2019s the same asterisk, by the way, next to the AWS free tier. But as you start elevating things into higher environments, you start to see gating around things like who has access to what, security reviews, et cetera, et cetera, and ideally, by the time you wind up getting into production, almost no one should have access and that access that people do have winds up being heavily gated. That is, of course, the vision that folks have. In practice, reality is what happens instead of what we plan on. The idea of it works in theory, but not in production is of course, why I call my staging environment \\u2018theory.\\u2019 Does that tend to resonate as far as what you\\u2019ve seen in the wild?


John: Yeah. Very much so. And when I joined the company, and we put together our [standard 00:11:11] pipelines for developers to be able to do everything, the rule that I would give to my team\\u2014so I manage a small team of cloud engineers\\u2014the one rule I would say is, \\u201cWe have access to prod because we need to provision resources, but when we\\u2019re going to build the pipelines for the developers, you have to build everything in such a way that the developers will only have read-only access to the production environment, and that is only to go and see their logs.\\u201d And at least try to foster this notion that developers do not need access to production, as much as possible because that avoids people going and do something they shouldn\\u2019t be doing in those production environments.


Now, as the pipeline progresses and applications get deployed to production, there are some operational capabilities that people need to have, and so in that case, what we do is we try to fine-tune what do people need to do and grant those people access to the accounts so that they can perform the jobs and I don\\u2019t have to be woken up at two in the morning. The developers are.


Corey: One thing that I think is going to be a cause of some consternation for folks\\u2014because I didn\\u2019t really think about this in any meaningful sense until I started acting as a consultant, which means you\\u2019re getting three years of experience for every year that you\\u2019re in the wild, just by virtue of the variety of environments you encounter\\u2014on some level, there\\u2019s a reasonable expectation you can have when you\\u2019re at a small, scrappy startup, that everyone involved knows where all the moving parts live. That tends to break down with scale. So, the idea of a Cloud Center of Excellence has been bandied around a lot. And personally, I hate the term because it implies the \\u2018Data Center of Mediocrity,\\u2019 which is a little on the nose for some people at times. So, the idea of having a sort of as a centralized tiger team that has the expertise and has the ability to go on deep dives and sort of loan themselves out to different teams seems to be a compromise between nobody knows what they\\u2019re doing and, every person involved should have an in-depth knowledge of the following list of disciplines.


For example, most folks do not need an in-depth primer on AWS billing constructs. They need about as much information fits on an index card. Do you find that having the centralized concentration of cloud knowledge on a particular team works out or do you find that effectively doing a rotating embedding story is the better answer?


John: It varies a lot, I think, because it depends on the level of curiosity of the developers quite a lot. So, I have a huge developer background. People in my team are probably more coming from ex-IT environments or this sort of operation and then it just naturally went into the cloud. And in my opinion, is fairly rare to find somebody that is actually good at doing both AWS and coding. I am by no means really, really great at coding. I code pretty much every day but I wouldn\\u2019t call myself a professional developer.


However, it does bring to my knowledge the fact that there are some good patterns and good practices that you can bring into building your applications in the cloud and some really bad ones. However, I think it\\u2019s really down to making sure that the knowledge is here within the team. If there\\u2019s a specialized team, those really need to be specialists. And I think the important thing then is to make sure that the developers and the people around you that are curious and want to ask questions know that you\\u2019re available to them to share that knowledge. Because at the end of the day, if I\\u2019m the only one with the knowledge, I\\u2019m going to be the one who is always going to be on call for this or doing that and this is no responsibility that I want. I am happy with a number of responsibilities, but not to be the only person to ever do this. I want to go on holidays from time to time.


So, at the end of the day, I suppose it really is up to what people want or expect out of their careers. I do a job that it was a passion for me since I was about 14 years old. And I\\u2019ve always been extremely curious to understand how things work, but I do draw the line that I don\\u2019t write anything else than Python these days. And if you ask me to write Java, I\\u2019ll probably change job in the flip of a second. But that\\u2019s the end of it. But I enjoy understanding how Java things work so that I can help my developers make better choices with what services in AWS to use.


Corey: On some level, it feels like there\\u2019s a, I guess, lack of the same kind of socialization that startups have sort of been somewhat guided by as far as core ethos goes, where, oh whatever I\\u2019m working on, I want to reach out to other people, and, \\u201cHey, I\\u2019m trying to solve this problem. What is it that you have been working on that\\u2019s germane to this and how can we collaborate together?\\u201d It has nothing to do, incidentally, with the idea that, oh, big company people aren\\u2019t friendly or are dedicated or aren\\u2019t good or aren\\u2019t well-connected; none of that. But there are so many people internally that you\\u2019re spending your time focusing on and there\\u2019s so much more internal context that doesn\\u2019t necessarily map to anything outside of the company that the idea of someone off the street who just solved a particular problem in a weird way could apply to what a larger company with, you know, regulatory burdens, starts to have in mind, it becomes a little bit further afield. Do you think that that\\u2019s accurate? Do you think that there\\u2019s still a strong sense of enterprise community that I\\u2019m just potentially not seeing in various ways because I don\\u2019t work at big companies?


John: It\\u2019s a very fine line to walk. So, when I joined the company, I was made aware that there\\u2019s a lot of Terraform and Kubernetes, which I went [unintelligible 00:16:28] all the way with CloudFormation is yes. So, that was one of the changes I knew I would have. But I can move an open mind and when I looked around at, okay, what are the Terraform modules\\u2014because I used Terraform with anger for an entire year of suffering\\u2014and I thought, \\u201cOkay, well, maybe people have actually got to a point where they\\u2019ve built great modules that I can just pick up off the shelf and reuse or customize only a tiny little bit, add maybe a couple of features and that\\u2019s, it move on; it\\u2019s good enough for me.\\u201d But as it turns out, there is I think, a lot of the time a case where the need for standardization goes against the need for business to move on.


So, I think this is where you start to see silos start to being built within the company and people do their own thing and the other ones do their own. And I think it\\u2019s always a really big challenge for a large company with extremely opinionated individuals to say, \\u201cAll right, we\\u2019re going to standardize on this way.\\u201d And it definitely was one of the biggest challenge that I had when I joined the company because again, big communities and Terraform place, we\\u2019re going to need to do something else. So, then it was the case of saying, \\u201cHey, I don\\u2019t think we need Kubernetes and I definitely don\\u2019t think we need Terraform for any the things\\u2014for any of those reasons, so how about we do something a little different?\\u201d


Corey: Speaking of doing things a little bit different, you were recently featured in an AWS Open-Source Roundup newsletter that was just where you, I think, came across my desk one of the first times, has specifically around an open-source project that you built: ECS Compose-X.


So, I assume it\\u2019s like, oh, it\\u2019s like Docker Compose for ECS and also the \\u2018X\\u2019 implies that it is extreme, just, like, you know, snack foods at the convenience store. What does it do and where\\u2019d it come from?


John: Right. So, you said most of it, right? It literally is a question where you take a Docker Compose file and you want to deploy your services that you worked on and all of that together, and you want to deploy it to AWS. So, ECS Compose-X is a CLI tool very much like the Copilot. I think it was released about four months just before Copilots came out\\u2014so, sorry, I beat you to the ball there\\u2014but with the Docker Compose specification supported.


And again, it was really out of I needed to find a neat way to take my services and deploy them in AWS. So, Compose-X is just a CLI tool that is going to parse your Docker Compose file and create CloudFormation templates out of it. Now, the X is not very extreme or anything like that, but it\\u2019s actually coming from the [finite 00:18:59] extension fields, which is something supported in Docker Compose. And so, you can do things like x-RDS, or x-DynamoDB, which Docker Compose on your laptop will totally ignore, but ECS Compose-X however will take that into account.


And what it will do is if you need a database or a DynamoDB table, for example, in your Docker Compose file, you do [x-RDS, my database, some properties, 00:19:22]\\u2014exactly the same properties as CloudFormation, actually\\u2014and then you say, \\u201cI want this service to have access to it in read-only fashion.\\u201d And what ECS Compose-X is going to do is just understand what it has to do when\\u2014meaning creating IAM policies, opening security groups, all of that stuff, and make all of that available to the containers in one way or another.


Corey: It feels like it\\u2019s a bit of a miss for Copilot not to do this. It feels like they wanted to go off in their own direction with the way that they viewed the world\\u2014which I get; I\\u2019m not saying there\\u2019s anything inherently wrong with that. There\\u2019s a reason that I point kubernetestheeasyway.com to the ECS marketing site\\u2014but there\\u2019s so much stuff out there that is shipped or made available in other ways with a Docker Compose file, and the question of okay, how do I take this and run it in Fargate or something because I don\\u2019t want to run it locally for whatever reason, and the answer is, \\u201cThat\\u2019s the neat part. You don\\u2019t.\\u201d


And it just becomes such a clear miss. There have been questions about this Since Copilot launched. There\\u2019s a GitHub issue tracking getting support for this that was last updated in September\\u2014we are currently recording this at the end of March\\u2014it just doesn\\u2019t seem to be something that\\u2019s a priority. I mean, I will say the couple of times that I\\u2019ve used Copilot myself, it was always for greenfield experiments, never for adopting something else that already existed. And that was\\u2026 it just felt like a bit of a heavy lift to me of oh, you need to know from the beginning that this is the tool you\\u2019re going to use for the thing. Docker Compose is what the ecosystem has settled on a long time ago and I really am disheartened by the fact that there\\u2019s no direct ECS support for it today.


John: Yeah, and it was definitely a motivation for me because I knew that ECS CLI version 1 was going into the sunset, and there wasn\\u2019t going to be anything supporting it. And so, I just wanted to have Docker Compose because it\\u2019s familiar to developers and again, if you want to have adoption and have people use your thing, it has to be easy. And when I looked at Copilot the first time around, I was extremely excited because I thought, \\u201cYes, thank you, Amazon for making my life easy. I don\\u2019t have to maintain this project anymore and I\\u2019m going to be able to just lift and shift, move over, and be happy about it.\\u201d But when the specification for Copilot was out and I could go for the documentation, I was equally disheartened because I was like, \\u201cOkay, not for me.\\u201d


And something very similar happened when they announced Proton. I was extremely excited by Proton. I opened a GitHub issue on the roadmap immediately to say, \\u201cHey, are you going to support to have some of those things together or not?\\u201d And the fact that the Proton templates\\u2014I mean, again, it was, what, two, three years ago now\\u2014and I haven\\u2019t looked at Proton since, so it was a very long time now.


Corey: The beta splasher was announced in 2020 and I really haven\\u2019t seen much from it since.


John: Well, and I haven\\u2019t done anything [unintelligible 00:22:07] with it. And literally, one of the first thing did when the project came out. Because obviously, this is an open-source project that we use in Sainsbury\\u2019s, right because we deploy everything in [ECS 00:22:17] so why would I reinvent the wheel the third time? It\\u2019s been done, I might as well leverage it. But every time something on it came out, I was seeing it as the way out of nobody\\u2019s going to need me anymore\\u2014which is great\\u2014and that doesn\\u2019t create a huge potential dependency on the company for me, oh, well, we need this to, you know, keep working.


Now, it\\u2019s open-source, it\\u2019s on the license you can fork it and do whatever you want with it, so from that point of view, nobody\\u2019s going to ask me anything in the future, but from the point of view where I need to, as much as possible, use AWS native tools, or AWS-built tools, I differently wanted every time to move over to something different. But every time I tried and tiptoed with those alternative offerings, I just went back and said, \\u201cNo, this [laugh] either is too new and not mature enough yet, or my tool is just better.\\u201d Right? And one of the things I\\u2019ve been doing for the past three years is look at the Docker ECS plugin, all of the issues, and I see all of the feature requests that people are asking for and just do that in my project. And some with Copilots. The only thing that Copilot does that I don\\u2019t do is tell people how to do CI/CD pipelines.


Corey: One thing you said a second ago just sort of, I guess, sent me spiraling for a second because I distinctly remember this particular painful part. You\\u2019re right, there was an ECS CLI for a long time that has since been deprecated. But we had internal tooling built around that. When there was an issue with a particular task that failed, getting logs out of it was non-trivial, so great. Here\\u2019s the magic incantation that does it.


I still haven\\u2019t found a great way to do that with the AWS v2 CLI and that feels like it\\u2019s a gap where yes, I understand, old tools go away and new ones show up, but, \\u201cHey, I [unintelligible 00:24:05] task. Can you tell me what the logs are?\\u201d \\u201cNo. Well, Copilot\\u2019s the new answer.\\u201d \\u201cOkay. Can I use this to get logs from something that isn\\u2019t Copilot?\\u201d \\u201cOh, absolutely not.\\u201d And the future is inherently terrible as a direct result.


John: Yeah. Well, I mean, again, the [unintelligible 00:24:20]\\u2014the only thing that ECS Compose-X does is create all the templates for you so you can, you know, then just query it and know where everything has been created. And one of the things it definitely does create is all of the log groups. Because again, least-privileged permissions being something that is very dear to me, I create the log groups and just allow the services to only write in those log groups and that\\u2019s it.


Now, typically this is not a thing that I\\u2019ve thought Compose-X was going to do because that\\u2019s not its purpose. It\\u2019s not going to be an operational tool to troubleshoot all the things and this is where I think that other projects are much better suited and I would rather use them as an extension or library of the project as opposed to reinvent them. So, if you\\u2019re trying to find a tool for yourself to look at logs, I highly recommend something called \\u2018AWS logs,\\u2019 which is fantastic. You just say, \\u201cHey, can you list the groups?\\u201d \\u201cOkay.\\u201d \\u201cCan you get me the groups and can I tell them on a terminal?\\u201d


And that\\u2019s it. Job done. So, as much as I enjoy building new features into the project, for example, I think that there\\u2019s a clear definition between what the project is for and what it\\u2019s not. And what it\\u2019s for is giving people CloudFormation templates they can reuse in any region and deploy their services and not necessarily deal with their operations; that\\u2019s up to them. At the end of the day, it\\u2019s really up to the user to know what they want to do with it. I\\u2019m not trying to force anybody into doing something specific.


Corey: I would agree. I think that there\\u2019s value to there\\u2019s more than one way to do it. The problem is, at some point, there\\u2019s a tipping point where you have this proliferation of different options to the point where you end up in this analysis paralysis model where you\\u2019re too busy trying to figure out what is the next clear step. And yes, that flexibility is incredibly valuable, especially when you get into, you know, large, sophisticated enterprises\\u2014ahem, ahem\\u2014but when you\\u2019re just trying to kick the tires on something new, I feel like there\\u2019s a certain lack of golden path where in the event of not having an opinion on any of these things, this is what you should do just to keep things moving forward, as opposed to here are two equal options that you can check with radio boxes and it\\u2019s not at all clear what you which does what or what the longer-term implications are. We\\u2019ve all gotten caught with the one-way doors we didn\\u2019t realize we were passing through at the time and then had to do significant technical debt repayment efforts to wind up making it right again.


I just wish that those questions would be called out, but everything else just, it doesn\\u2019t matter. If you don\\u2019t like the name of the service that you\\u2019re creating, you can change it later. Or if you can\\u2019t, maybe you should know now, so you don\\u2019t have\\u2014in my case\\u2014a DynamoDB table that is named \\u2018test\\u2019 running in production forever.


John: Yeah. You\\u2019re absolutely right. And again, I think it goes back to one of the biggest challenges that I had when I joined the company, which was when I said, \\u201cI think we should be using CloudFormation, I think we should be using ECS and Terraforming Kubernetes for those reasons.\\u201d And one of the reasons was, the people. Meaning we were a very small team, only five cloud engineers at the time.


And as I joined the company, they were already was three different teams using four different CI/CD tools. And they all wanted to use Kubernetes, for example, and they were all using different CI/CD\\u2014like I said, just now\\u2014different CI/CD tools. And so, the real big challenge for me was how do I pitch that simplicity is what\\u2019s going to allow us to deliver value for the business? Because at the end of the day, like you said many, many times before, the AWS bill is a question of architecture, right? And there\\u2019s a link and intricacy between the two things.


So, the only thing that really mattered for me and the team was to find a way, find the service that was going to allow to do a number of things, A, delivering value quickly, being supported over time. Because one of the things that I think people forget these days\\u2014well, one of the things I\\u2019m allergic to and one of the things that makes me spiral is what I call CV-driven tech choices where people say, \\u201cHey, I love this great thing I read about and I think that we should use that in production. How great idea.\\u201d But really, I don\\u2019t know anything about it and is then up to somebody else to maintain it long-term.


And that goes to the other point, which is, turnover-proof is what I call it. So, making tech choices that are going to be something that people will be able to use for many, many years, there is going to be a company behind the scenes that he\\u2019s going to be able to support you as well as you go and use the service for the many, many years to go.


Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where\\u2019s the best place for them to find you?


John: So, people can find me on LinkedIn. I\\u2019m also around on Twitter these days, although I probably about have nine followers. Well, probably shouldn\\u2019t say that [laugh] and that doesn\\u2019t matter.


Corey: It\\u2019s fine. We\\u2019ll put a link into it\\u2014we\\u2019ll put a link to that in the [show notes 00:29:02] and maybe we\\u2019ll come up with number ten. You never know. Thanks again for your time. I really appreciate it.


John: Thanks so much, Corey, for having me.


Corey: John Mille, Principal Cloud Engineer at Sainsbury\\u2019s. I\\u2019m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you\\u2019ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you\\u2019ve hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you go to great pains to type out but then fails to post because the version of the tool you use to submit it has been deprecated without a viable replacement.


Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

'