Hedgehog is not an out.
It's not a thing you do.
It's a way that you do something.
It's agility.
And so you do things in an agile way.
It's a really difficult problem because everybody wants to be told what to do.
When it's scary, when it's unknown, people are looking around for that figure that says, don't worry, I've got it covered.
Do it this way. It'll be fine.
Hello and welcome to episode 44 of the Metacast Behind the Scene Podcast.
This is Arnab and with me, I have my co-host, Ilya.
We are the founders of Metacast.
And today, we have an amazing guest with us, perhaps the biggest influence in my life as a software engineer
so far. And say hello, Dave Thomas.
Hello, Dave Thomas.
What was the kind of you to say?
I'll let you introduce yourself in a little bit, but I just want to start with, like I said, the influence that you have had in my life and career, back when I started at Amazon as an SD, like an entry-level engineer.
One of the engineers in my team suggested that I read this book, the pragmatic programmer.
You've done so many renowned things, but this is the thing that, like, immediately, I think, connect with you.
And this book, aside from all the best practices and the teachings from it, I think it fundamentally changed the way I approach software development in that I always need to be a craftsman.
I always need to be practicing.
I think some of that goes into us creating this company, Ilya and me, because after getting promoted and promoted and promoted, you get away from actually doing the craftsman things, right?
And after I think getting into that habit, I was not happy with that eventually.
And so, Ilya and I decided, let's go build something else.
So that's kind of like the foundational things that you have had in me, but your book also helped me evangelize unit testing, CI, CD infrastructure, all of these best practices early on at Amazon and elsewhere too now.
So thank you for it.
I have the book right here with me.
This is a 2009, I looked it up, and that's kind of like when I read it the first time, so it makes sense, yeah.
Cool. Of course, that makes me feel very, very old.
But apart from that, that's cool.
Yeah, so please introduce yourself to our listeners.
Well, my name is Dave Thomas.
I'm English, but I've been living in the US now for 30-something years.
My wife is American.
I started off programming 1972, I think.
And I have probably programmed just about every day since then.
Fundamentally, I enjoy it.
It's kind of like my happy place is writing code and discovering things along the way.
So I had a pretty standard kind of career.
I did computer science at university in London.
I was doing a PhD program and I got recruited away by what nowadays would be called a startup.
Back then we called them sweatshops.
It was the best experience because this was a company that was formed to write custom software for people.
And because of the fact they were small and they were just starting to grow, they said yes to everything.
The range of different industries and project types and people that I got exposed to was phenomenal.
And the assumption was yes, we can do it.
That kind of left me in this position where I really didn't have any excuses.
I couldn't say well, you can't really do that because we'd committed to do it.
I learned how to basically get things done in code.
And that was just absolutely fantastic.
And I stayed with them for a while. I moved across briefly to what was at the time.
I think the United Kingdom's only surviving computer manufacturer.
And I joined their kind of research place.
It was called CTL or ITL depending on the year.
And they produced a media computer, which was quite an old fashioned architecture.
But it was cheap and it was made in the UK, which meant that for various government contracts and defense contracts and everything else it was important.
But it also meant we got some pretty weird stuff we're asked to do.
One of my most fondly remembered projects was we got a job with the European Space Agency writing or supplying the system to do the on the ground checkout for the Geotosatellite that went to see Halley's comet.
That involved some bespoke hardware, a very, very wide, very high speed parallel interface.
There could suck data off all of the various test points and record it, which meant changes to obviously the hardware, but also to the operating system.
And we had to write some really pretty cool code to intercept it.
And there was micro code running on the actual controller itself.
And the high spot obviously is we got to go across to Holland and get our stuff working in the test lab, which is where everybody wears the bunny suits and it's all white.
And you have this sort of golden thing in the middle, which is the satellite.
That was good fun. That was very good fun.
What's here was it?
Oh, now you're asking.
I can't remember early ish 80s, I think, but I'm sure Google will prove me wrong. Don't ask me dates. I have no idea about dates.
But did that for a while? Then I formed my own company in the UK. Well, actually, that's not quite true.
A friend from this company that we just talked about and I as a side project developed a terminal emulator.
Like as an open source project or.
Now, I mean, the idea of a public internet was still five years away.
Opera existed and there were universities that talked onto universities.
But we downloaded software using dial-up modems and we were connected services like Compuserve.
Look it up folks. Look it up.
So no, I'll open source.
But no, this is a product and it's claimed to fame. It ran on digital equipment, microse and it also run on IBM PCs.
And it's claimed to fame was you could actually connect to multiple computers at the same time and multiplex them onto one screen.
That found quite a big market in the financial marketplace where real estate is really, really tough.
So this got to the point where it actually became viable. So we formed a company and we sold that.
And then we started writing software for people. That was really good fun.
Again, we were in that lean hungry stage. So we would say yes to everything. And so we got to do some pretty wild things.
Along the way accumulated like basically experience working on all sorts of different projects and styles.
Absolutely, absolutely. Also kind of like learning what not to do.
It's a lot easier to see things go wrong when it directly affects your bottom line.
And so we got very good rapid feedback when we were doing things wrong. And that was that was useful.
So at some point we became the UK point of contact for a US computer manufacturer called Stratus.
And what they had was a large mini computer that was fault tolerant.
Everything was duplicated. Everything ran in parallel. And you could go up to it and like pull boards out.
They would carry on running with that interruption, which is kind of cool.
And so we started doing work with that again, a big market in the city of London.
And they sent across their a sale support person to help us.
And this sale support at some point went back to the States.
And a few years later I'd also left the company at this point.
And I was doing independent consulting. And he phoned me up and said, Hey, look, I'm now working at a financial company.
We're doing switching of debit cards, credit cards.
And I promised them that we would rewrite all of that switch software in a camera.
It was like six months or eight months or something.
And I said, yes, he said to me. So I said, of course we can do that.
And then I said, then I had to call you to make it happen.
So I flew across and we had a look at it.
And I said, I'm going to need some help.
And he said, well, I had a friend from college who's independent too.
And that's when I met Andy Hunt.
So Andy and I spent probably in the end about 10 months.
The office they gave us was a converted closet, no windows, no air conditioning.
We were working in Atlanta at the time.
So it was a very close experience.
But we did it. We got the switch working.
And we kind of enjoyed working together.
And so we kept doing it.
And for the next five, six years, we did that.
One of the things we noticed that all of our clients tended to make the same mistakes.
So they were not testing this offer before they shipped it.
They weren't doing version control.
They didn't have any automation. All of these kind of things.
So we started jotting down some notes.
And the idea was that when we went to a new client, we were to give them these notes as kind of like a baseline and say, if you look at this, that's kind of what we're going to be talking about.
And as things tend to do, it started out as being, it'll be like five pages.
And then we got to 50 pages.
And my wife said, maybe you should turn this into something.
And we were like, nah, and she was she insisted.
So we thought we'd never done anything like write a book before.
And we had no idea what it meant.
So we came up with a really clever plan.
We would take what we had and we would send it to the publisher.
We thought was the best technical book publisher in software, which at the time was Addison Wesley.
And we said, okay, we'll send this to them.
And they will say no. But as a professional company, they will tell us what's wrong with it.
And then we can fix it.
And they foiled us.
You know, they totally ruined our plan because they said yes.
And so we ended up writing a pragmatic programmer for them.
At the time, we had already created like some tools to help us write something in book format.
And so they reluctantly agreed to let us use our own tools.
And so we do all the typesetting and layout and the indexing of the first edition that one that you showed.
And from that point on, we kind of got the idea that, hey, writing books is kind of fun.
And we already had tools to do it.
And so after a gap of about a year, we did the Ruby book.
Or I did the Ruby book.
We brought in Mike Clark to write a book on automation.
And we suddenly realized we had a company.
And at the time, the consultancy business was pretty dire.
It was just after the Y2K thing.
And people didn't have budget.
And so there wasn't that much work going around.
So we decided to focus more on the books for a while.
And that just went from strength to strength.
For a long time, the only place I would like go to to look for technical books was pragmatic programmers.
Because I knew that I don't have to like read reviews of this stuff.
Whatever is on here is golden.
Oh, that's kind of you.
It was fun.
It really was fun.
And I was lucky because I had a fair amount of contacts from doing conferences and this kind of stuff.
You know, if I saw someone speak at a conference and I thought, oh, that's interesting.
I would typically go and say, hey, do you want to write a book?
And a number of them were silly enough to say yes.
And so we actually got some really good authors and some really good titles out of that.
I was in one of your, I think it was the Ruby Conf in Australia.
Maybe 2013 or 2014. I was in one of your elixir.
I think at that time you were teaching elixir and you were writing the elixir book.
So I was in one of those classes, yeah.
Oh, okay. Cool. Cool. That was always good fun.
So this book, 24 years later, we look back at this book as one of the seminal works in the software industry, right?
Because a lot of the things that you talked about, like you said, believe it or not, people were not writing tests.
There was no definition of what a unit test is versus what an integration test is.
What should different things look at?
Automation, observability, and those things have become like standards now.
You don't have to influence anybody to go do these things anymore.
You're still consulting.
What are the new things that you're seeing that, okay, I need to again repeatedly go back to my clients and tell them to do this or do that.
What are those new things that you're seeing now?
So I got to tell you first of all, just in fairness, that I'm not actually being paid to consult.
I'm kind of busy right now with the bookshelf, but I am talking to a lot of people, many, many people.
And little bit of history, if you don't mind.
Back in 2000, software was in a really, really, really bad state.
The vast majority of projects never completed.
Even fewer projects completed successfully and actually delivered what they said they would.
And when I say the vast majority, I mean, it's like more than I think 60-something percent failed.
This was using 1990s technology, which was nowhere near as complicated as it launched today.
So people were responding to that by creating systems frameworks in which you could develop software.
I don't mean like software frameworks, but like structural project frameworks.
And everybody in their dog had its own framework that you were supposed to use.
And none of them worked.
And the reason that none of them worked is that they made a very, very faulty assumption.
And that is that people know what they want.
And people don't know what they want.
Nobody knows what they want.
They may think they know what they want.
But you give it to them and they'll go, oh yeah, but that wasn't quite what I meant.
And when you're talking about software that takes a year to write, that's a big investment to discover that you actually didn't quite want what you said.
So the motivation behind the manifesto of fragile software development was to try to find a way to break that logger.
The realization is that the only way to make any progress when you are not sure of what you're doing is to use feedback.
That's the only way you can do it.
Literally, it doesn't matter.
I don't care if you are walking across a desert or whether you are flying an airplane or whether you are writing a brand new podcast application.
The only way to do it is to collect feedback and to collect feedback at every single level.
All the way down from the lines of code, all the way up to the how are we doing as a business?
I can very small increments.
That's really flies into that right.
So if you're going to collect feedback, the longer you leave it to collect the feedback, the more the error potentially.
And the bigger the correction and a correction is a cost.
So the more you can get feedback rapidly, it's a better.
Now, this obvious time scales for different kinds of feedback.
So feedback to do with, say, lines of code, you can get pretty much instantaneously, if you want right-align a code or two, and then try running it against the test.
And you can get feedback pretty quickly.
Some feedback, like going to the client and saying, is this what you meant?
Well, that might take a day or two.
So that's kind of like on a different level.
Some feedback on things like, how did we do as a project team?
What would we do differently?
What did what went well?
Well, obviously that relies on the length of a project.
So there's different scales of feedback.
But the rule is, never start something until you know what feedback you're going to collect.
Always try to minimize the time it takes to get that feedback.
So basically, the feedback loop should be part of the structure of the project.
Feedback loop is the structure of the project.
In the ideal world, the entire project is structured purely as a feedback machine that just happens to deliver software at the end of it.
But here's the thing.
In a successfully orchestrated team, it doesn't even have to develop a software.
A client can come to you and say, I have this need, and the team can say, oh, you know what, we can follow that.
Without any software, we could just like say, what do you do this, this, and this?
And the client go, oh yeah, that works.
And they're successful.
That's a successful project.
So the feedback is what drives what you do.
And one of the things that I am discovering back in 2000, the idea was you could write your requirements document.
And that requirements document was then sacred.
No one could change it.
And that's what the team would develop to deliver against.
And that's what the the manifesto said was not true.
The manifesto said, people don't know what they want.
So we have to develop incrementally, show them and get feedback from them and fix it.
But now 20 years later, we are falling back into the same problem.
We're talking about having, you know, liais on people, different people call them different things, product managers or whatever else, right?
And these people are now the repository of what the client wants.
And they will come down with documents that say, this is what's needed in this iteration.
Or this is what the client wants in this iteration.
And so we form back exactly into the same trap of thinking we understand the requirements we don't.
In some ways, I think maybe without those people or those defined roles, but this has always been a problem.
That the fact is that you can define the problem before starting to do anything about it.
That's the falsehood.
Absolutely.
Well, it's based on two things, I think.
First of all, in many facets of quote engineering, that is true to some degree in that I can go and I can find the properties materials and I can measure the distance.
And based on that, I can slap together a standard design bridge.
And I'm pretty sure that it won't fall down.
The problems in classical engineering come when somebody says, yes, but can you do it cheaper?
But that's it, the whole point actually of engineering.
Anybody can build a bridge right?
If I wanted to build a bridge across a river, I just dump dirt into it until you can drive across.
Anyone can do that.
But an engineer can do it cheaply and allow the water to flow and boats to go into the ether and the like and stuff.
And so engineers can do it according to a template very, very simply.
The assumption is that we can do the same thing in software because we call it software engineering.
And we can't.
There are no tables that you can look up to tell you anything to do a software engineering.
Every project is different.
Every environment is different.
Every tool set is different.
Every developer, bless them are different.
And so we are not doing engineering, but the assumption was we were.
So that's one problem.
It's a problem, but it's also one of the beauties of software development because I am a civil engineer by education.
And after a while, I figured out that I really enjoy the creative side of software development.
Yeah, absolutely.
But it comes at a price obviously.
Because as software engineers, we have a very hard interface on one side, which is the computer.
And by heart, I mean, unchanging.
So, you know, we have to write something that eventually will run on this machine.
And we have this very ill-defined thing that people want us to do.
And our job is to sit in the middle and basically make intelligent guesses to make that happen.
It's like juggling with jello or something.
It's a mess.
And the fact we can do it at all given our limited understanding is seriously impressive.
But by doing some of the stuff in the Agile Manifesto, we can reduce some of the risk because if we reduce the time to get feedback, then we can more rapidly work out if we're doing it correctly or not.
If we encourage proper prototyping and proper experimentation.
And if we allow people to say, I made a mistake without punishing them.
If we do all of those things, then we're in a better position to deal with this uncertain world.
But many large companies do not have the flexibility to be, quote, agile all the way up.
So at some level, and typically it's the middle management, if you talk to really senior management, they'd love the idea of agility.
They'd love being able to say that you'll get stuff out there as fast as possible.
They don't like the idea of two-year plans.
But middle management lives and dies by their annual budget and their annual forecast.
And what kind of stuff?
And so middle management wants to impose that kind of engineering structure on their software team.
Because of that, and because of that kind of impedance mismatch between the management and the best way of running the teams, a whole bunch of companies have said, hey, we have the solution.
And we have techniques that will work that allow you to run agile in your company.
And because it's a big company, guess what? They've got a lot of money.
And so they'll give you $5, $10 million for the magic secret source that will turn your company agile.
But does it work?
No, of course not.
And it can't work because by the definition of agility, what works for one person will not work for another person.
If I went and I found the Olympic figure skater, and I recalled it every single movement that figure skater made, and wrote it into a book, and then gave it to somebody else and said, here, do this and you'll be an Olympic figure skater.
Would it work?
Obviously not.
That's exactly what people are doing with all of these men.
That's exactly what people are doing with all of these methodologies that are supposed to be agile.
And unfortunately, people are believing it.
And so nowadays, I got really depressed when I looked around and I saw they call it agile now.
And I got to admit every now and then I fall into it.
agile is not a noun.
It's not a thing you do.
It's a way that you do something.
It's agility.
And so you do things in an agile way, but these people aren't selling that.
These people are selling, do this and you are agile, which is totally bogus.
Yeah, there's something in methodology.
Yeah, yeah.
And there's lots of different levels of it too.
It's a really difficult problem because everybody wants to be told what to do.
When it's scary, when it's unknown, people are looking around for that figure that says, don't worry, I've got it covered.
Do it this way.
It'll be fine.
But as human beings, we seek to create templates for things because they're predictable.
And it's the predictability that we like.
Liquid and illusion of predictability, rather.
We are.
But I mean, also it's the illusion that it's somebody else's responsibility now.
It's kind of like the Nuremberg defense.
I was just following orders and people look at that in a corporate setting.
And a lot of these companies were people impose these frameworks on top of them.
The project teams kind of like go, well, this isn't going to work.
But that's okay because now it's not awful.
In the past, people used to blame us.
We've delivered what you asked for.
The fact that it doesn't work is not our problem.
Yeah. Yeah. I don't care if the whole Sony has three legs.
That's what you asked for.
So anyway, that got me really, really oppressed.
And so I started going around recently, actually, to try and take the temperature of what people were actually doing.
So I put out a call and I've had maybe a hundred people sign up to chat with me over Zoom.
And I'm about halfway through that right now.
And it has been absolutely fascinating.
I would say maybe one in three of the calls were from people who were in a bad situation.
Where they were suffering from some of the kind of things we've talked about.
Or the classic bad manager or evil manager or whatever it might be.
But the majority of people were doing what developers have always done.
And that is they found ways to beat the system.
And they found ways to get their job done.
Even within the confines of some of these really bad practices.
And what they're doing is really interesting.
I'm still kind of at the early stages of picking it all apart.
But there's a couple of threads that are coming out consistently that successful and happy teams are all doing.
Do you want to know what they are?
Yes. So one of the things they're doing is once a team gets to a certain size.
And that's probably about six to eight people.
Or once a team starts to have some teams or once a project starts to have some teams.
One of the practices of successful teams is that they have at least one person, senior person, whose job is not to be on a team, but whose job is to basically float between all the teams.
And just be chatty, be social to find out what people are doing, what problems they're having.
Where people are duplicating effort between teams, where people are misunderstanding what other people are doing, all that kind of stuff.
And this person's job is basically to act as a lubricant between all the different things that are going on.
And to make sure that things are not rubbing up against each other the wrong way.
They are looking at people who may be struggling and saying, what can we do to help these people?
One of the guys I spoke to was kind of interesting. He instituted a cross team pairing program where for like a conference like one afternoon, or something, you had to pair with someone on a different team.
He then tried to arrange things that say you had a developer that was struggling with racial databases.
He would find somebody on another team that was absolutely nailing it and say, okay, I'd like you to go pair with Fred over there for an afternoon.
And see if you can work out where his problems are and help him.
And doing that, you got to know what other people were doing. And you actually helped what other people were up to.
And Fred then knew that you can go and talk to Sylvia or whatever her name was.
And ask questions in the future. So that kind of role. I think that is often misunderstood.
People sometimes call that the project lead or the project manager.
But those roles are telling people what to do in a way.
And the role I'm talking about is asking people what needs to be done.
I think that's in a way similar to the principle engineer role at Amazon.
I think the thing with the principle engineer role at these big companies is that it's so many things.
But this is definitely one of those things is you are talking to people.
Maybe sometimes they're struggling or they're not at an individual level to different engineers and trying to figure out what's working.
What's not and you're creating those bridges between teams or essentially you are the bridge between different teams.
I think Apple has a role that's similar as well.
I may misunderstand this, but I believe there's an Apple role called staff engineer that does the same kind of thing.
And it's significant to me that companies that are successful have those roles.
That's an important one, I think.
The other thing that seemed to be important is giving the developers access not to what customers want, but to what customers need.
It's kind of the difference between what you say and what you mean and different companies had different ways of doing it.
One company had this thing where when you kick a project off, you always bring either the team to the customer or the customers of the team.
Ideally you take the team to the customer and you actually don't just sit in the meeting room with their manager, but you actually go and talk to the people that this work is for because quite often the manager will have one idea.
But the people I should do in the work will have a very different idea and you talk to them and you basically say, hey, good news.
We've got like infinite amount of money to write stuff for you.
What would you like? What would you need?
And obviously you're guided by what the project is about, but you're not bound by that.
I mean, I had an example of that once on a project where I went, one of my kind of prime sources from information was one of the people who was working on the help desk for these particular kind of transactions.
Because every now and then I'd have to go ask, what does this mean and what does that mean?
And one time I watched and she was sitting there and she had a pad of paper on the side and she'd be talking to a customer and she'd bring a screen up and she would write a 14 digit number down on her pad of paper.
Then go to another screen and type this number in.
One of the core ended I said, what are you doing? Why are you writing down this 14? Oh, because it's actually the same system, but these two screens don't talk to each other.
And those are the kind of things that a manager might not notice, pick up on or whatever when they're specifying a system.
So by having the developers out there and talking about need and not about requirements, I think is really, really important.
Because the other thing you can do if you understand need is you can deliver against it.
You can prioritize it.
And so if you understand what the customers needs are and what the priorities are, you can deliver incomplete projects that actually will help iteratively and also kind of speculatively in that you can deliver it.
And there have been times where I delivered something in that kind of vein.
And the customer said, hey, that's good enough. That's what we needed.
And all the rest were just kind of icing on the cake, but we don't need that.
I was curious on your thoughts about, you know, once was in a project where the system was perfect, but once I spent the night with people in warehouse, I realized, well, they just don't want the new system.
Because there's fear, this latent fear that managers don't recognize.
Actually, in this particular case, the guy was the manager. And basically, the whole project was almost sabotaged by that fear.
I'm curious if you have thoughts on how you do with that because it's all, I don't know what the right word to express. It's like latent fear.
Is the fear there because they're being measured against some productivity or something like that and they left to learn a new way of doing this in a new system? What is the root of the fear?
So the context here is logistics company. It's a customs clearance process. People have been doing the same thing over and over again, tens of times every night to clear customer shipments.
And then you introduce, with the new system comes a slight change to the process too.
And I think with people who just repeatedly do the same thing over and over, there is maybe just fear of new.
But as a spectator could also be fear of job security because if you can do things more efficiently, maybe fewer people are needed to do it.
The other thing I've personally found in those kind of systems is that as developers, we love looking for patterns.
And so we look at a system like that and we go, okay, so we can have a look at, I don't know what the details are, but this number and this number.
And that will tell us what bin it goes into and then this will tell us what form we need and what's kind of stuff, right?
And we'll then design a system based on this really beautiful structure that we put in place.
But the real world is never beautiful. And I bet you that the people on the ground handling all of those items, no 1000 special cases that they have to deal with every day.
And one of their fears is that unless you work that job for five years, you won't know about that.
And that we'll produce for them a system which basically doesn't work because of those edge cases.
And fixing manually all of those edge cases is going to be way more work for them than they ever had before.
And I've seen that happen quite a lot where we think things should be simple, we make things simple and we have this kind of arrogant clearly everything in the world has to be a hierarchy and everything has to follow rules and it doesn't, it just doesn't.
I always think that the best example of that is the world's most popular software, which is the spreadsheet.
The magic of the spreadsheet is it doesn't tell you what to do.
True, that's very true. It's very flexible. You can run cache balances or you can do like like you were talking in the other podcast 2015.
I think those book publishers they end up dating about books and spreadsheets.
Yeah, there's limits.
No, but I mean seriously, I mean what you're doing with that and with so many other devices is that you're enabling people to do things the way they want to do them.
And I would like to see us as an industry working on ways we can do something like that.
And I think actually to be fair, we're not doing too bad of a job right now.
I mean, if you look at the fact that you can now do on your phone stuff that you used to have to have a PC and a screen and everything else.
But nowadays you expect to be able to cut and paste something from here to there and have its format adjust automatically as you do it.
So again, you're in control. You're telling it what to do and it's working out how to get it done.
That is also something that I'm seeing. Is this move from specifying a requirement to meeting a need?
Yeah, and AI probably also plays a role here because it well in theory and also I guess that later stages of evolution it can better predict as to what you want.
And actually sometimes when I type an email in Gmail, I'm impressed how well it predicts simple phrases that I don't have to type anymore.
I just press a tab and just complete it for me.
Yeah, as an experiment, I tried to I was doing some layout work and I needed like three or four pages of sample text.
So I actually went to kind of one of the AI's and I said, write me 2000 words about this.
And I swear I could have published that and no one would have known. It was just scary, scary, good.
I mean, there is a lot of content right now that is being published that way.
I have a question about our startup. I'm almost a bit afraid to ask you this, but I'll do it still.
So we are at a very early stage of our company. We have some early customers. We are validating our idea. We're seeing starting to see some validation that yes, these needs are there.
And we're trying to figure out the way that we're trying to satisfy those needs is that a workable thing or not.
What I'm trying to say is this is very early stage. So if you had told me that I would be writing like a persistent without integration tests and things like that from scratch, I would have laughed at like whoever said that are now is going to do this like five years later.
But here we go.
Yeah, we go because I was in a very structured environment in AWS inside Amazon with lots of resources, lots of resources. Yes. How do you decide to balance between like the pragmatic way that we're trying to do this.
And at the same time be future proof. Oh, you can't be. It's impossible. Okay. There. First of all, the concept of future proof is about as crazy as the idea of perfection. How can you possibly be future proof unless you have some secret magic crystal ball that nobody else on the universe has.
There is no such thing as future proof. All you can be is easier to change than the next guy. You cannot predict what is going to happen. All you can do is say think of some bad cases, bad scenarios and say, okay, what happens if that happened.
Now am I going to structure my code so that that change is not going to put my company out of business. I watch people who will spend time putting in place three or four abstraction layers in their code just in case this happens just in case that happens.
This is code that they're going to throw away. This is their first prototype. This is their get off the ground validate the ideas code. This is code whose entire job is to get out there, let people try things crash every now and then but give you the information you know to make the next one better.
There is no point in future proofing that code as a startup, I can pretty much guarantee that you will be rewriting that code. The trick will be to make sure that you don't lose anything along the way.
So you need to make sure that the data formats you're using are not proprietary to a particular technology. You need to make sure that if you had to switch persistence, for example, then you haven't architected yourself totally around a unique feature of one persistence layer that others don't have.
But apart from that, I don't think you have to think about future proofing at this stage, at this stage, what you're thinking about is how do I get stuff out there? How do I start making money from it? And how do I learn what I have to do when I do it properly?
If you think about it right now, when you say future proofing, that's exactly the same as saying, I know exactly what I want.
Or what I'm going to want in a year. Well, what I'm going to want. Yeah, exactly. And I'm sorry to be the bearer of bad news.
One of the interesting thing here is we've made a commitment to build our system on a no SQL database. But then it also means sometimes I'm like, can you just show this?
But then it requires joining two tables, which you can't in no SQL, right? Instead, you have to make either multiple requests or you just don't do this feature at all because it's too expensive, right?
But then if you would go with the latest relational database, we know that maybe we reach like 10,000,000 users and it will start to break. And then we'll have to shard it or we'll have to re-architect everything.
And the most logical simplest way to build the thing if you were optimizing for speed only would be just use postgres my SQL and validate it.
But then we would have to rewrite the whole thing. But the problem is relational doesn't translate the right into no SQL. So it's a kind of a decision you have to commit to.
Or you commit to the rewrite and that rewrite will take the amount of time that it will take because it's going to be a significant rewrite or you choose the current simplest path, which probably is postgres, but I mean whatever.
I don't care if you use no SQL or SQL database, you choose the path that seems easiest to you right now and that will get you up and running.
Frankly, if you get to 100,000 users, then you should be happy that you have to make a change because of that. Not sad.
You get something that's up and running, but you say to yourself, OK, I know that should we become really successful. This is going to be a bottleneck.
But if your systems are anything like the systems I've worked on, the kind of queries that get you in trouble are a very small subset of the overall.
Most of the system is straightforward, go and find the user record, go and find the podcast for this user. And those are things that you can do equally well,
relationally or no SQL when you're doing the complex stuff, quite often it's in limited circumstances like reporting or you're setting up new accounts or something like that, you know, where you actually have to do the complex joins.
So maybe that would say, OK, as I'm developing the code, I'm going to be cognizant of the fact that this is not going to work as I get bigger either way, whether it's on the no SQL SQL.
And so I'm going to make sure that I keep this isolated or at least in its own little source tree or something and then come the place where suddenly you appear at the top of hack and news and you got a million subscribers.
Then you get a very uncomfortable week when you transition that one piece of code to use some kind of cash, no SQL cash or SQL cat, whatever it might be.
And then you move on, but I have that problem, I find myself constantly not starting projects because I'm spending all my time thinking about what could go wrong and how I could fix it.
And I just I have to keep telling myself, stop it, get something working. I understand what you're saying. And I know it looks scary, but I think you can mitigate the risk without actually going over the top at this phase.
So you're saying it's also it doesn't have to be binary choice. It's never a binary choice. The only people that will give you a binary choice is a sales person.
Yes, right. There are no binary choices in the world. Any decision worth making is nuanced.
So this has been super exciting. We are already like almost an hour into it. And I know Ilya, you wanted to go into the book and the identity that's tied to the book and that sort of side. Maybe this is a segue into that.
Yeah, I actually want to ask a question. So yesterday on LinkedIn, yeah, my depose, I said, we are talking to Dave Thomas tomorrow. These are the things that we want to talk about.
Does any of you have any questions that we want to ask? So our friend, Goaang, Hi, Goaang. He's a host of software, Mr. Adventure's podcast, which is a great podcast.
So he asked, is there something in the book that you didn't expect to still be the case after 24 years, but it's still true today. I think you thought about this for a while. It's a very good question.
The opposite question is easy. You know, other things now that don't apply. The answer is compared to the original. Yes, very much so.
There's also interesting question where the stuff we had in the book has been misunderstood. I guess we didn't explain it well. And therefore people have tended to misunderstand things.
The biggest example of that is the don't repeat yourself idea, dry to my absolute amazement, dry has become like a standard term in the industry.
But people quite often apply it wrongly. They apply it to the idea of effectively copying and pasting something. Don't repeat yourself. Don't copy this code and put it over there. But really it doesn't mean that.
It means that you should only ever have one implementation, particular concept, because that gives you one place to change when the world changes.
A repository process or information. Well, now I wouldn't go that formal. I would just, it's something where if you discover or if you're writing code and you find yourself doing the same kinds of things, ask us or an abstraction that would make it easier.
That was a surprise that we didn't get that right. And so the wording in the dry section in the new book is quite a bit different.
Like something that you thought would not be future proof, but it ended up being future proof.
There's a couple of things that people still aren't doing that are disappointed at. Nothing like major, nothing like headline.
One practice that I believe in really, really strongly is the idea of the engineering daybook where you keep a physical book at your side.
Yes, I remember reading the chapter. Yes.
Yeah, it is to my mind. It's my life line. I get so much benefit out of doing that. And we still had to emphasize it. And people are still not doing it.
In fact, people are doing it less. And people are doing things like, oh, it doesn't matter. I can stick it into notion or obsidian or whatever I'm using.
And it's not the same. One of the bad things now is so many of the new programmers have never actually been taught how to write cursive.
So, you know, it's really hard for them to take notes, but honestly getting a decent notebook with high quality paper and a nice pen makes you respect what you're doing so much more.
It makes you think about it. And the act of writing it down, slowing down gives you this time to reflect on how that interacts with all the other things you've written down.
And I don't know about you, but when I'm writing my daybook, I'll be writing notes. And then suddenly I'll draw a circle round of it and then a line that points up to like three or four items earlier on, which previously I'd made no connection.
So I am a really big fan of daybooks. And I'm also really, I got this pen, which is really cool. It's a fountain pen that's retractable, but it's not this pilot made one, which is ridiculously expensive.
It's like 150 bucks or something. And this one was like 30 bucks. And it uses the same nib. You know, it's the pilot uses. I love it. I keep this with me all the time.
I'll have to insert a plug here too. So I used to have a pilot pen. I just still have it. I don't like it. But I have this carbon fiber pen. The fountain pen too.
I mean, it's retractable. It's a regular pen. It's 15 bucks on Amazon. But because it's carbon fiber, it's very light, but also like it doesn't get cold, unlike metal pens.
It feels like metal. I guess it doesn't feel like plastic. I tried the metal pen to but metal pen gets too cold in the room.
Yeah. Yeah. This is just plastic, but it's fine. You know. Okay. So while we're geeking out on pens. Oh, I don't have it with me.
There is a travel looking pen by a German company called Koweko. I think I saw us pronounced, which is about half the size of this pen is about that big.
You unscrew it and then move the other end here. And it makes a full size pen. Absolutely fantastic. Nice. So I keep that in my pocket.
Coming back to this, something from 24 years ago, I kind of see that a lot of the things that we talked about have become day to day practices. Like nobody would question you if you want to say, like I want to set up a integration testing mechanism in here.
Everybody understands the value of it. The one thing I think we already talked about this, but I do want to reiterate.
I think we still haven't fully embraced the agility of doing things because to this day, we are still going backwards from when do we need to deliver this project on and try to try to shoehorn in the process after that.
And I don't have a good way like we struggle with it, Ilia at lots of places that you and I have worked at together separately, but this is a problem. And I don't have a good way of how to solve this because there is a definite need of projecting out when are you going to be able to release this thing.
I agree 100% I think that that is one of the big unsolved areas of agility that it is very, very hard at the interface between the agile world and the non agile world.
And I don't know anybody that's actually come up with a universal solution for that, which I did, but you're absolutely right.
Okay, so let's differentiate, I guess the pragmatic stuff and the agile stuff, the agile manifesto never was intended to be a set of practices that you had to follow.
All it really is is a set of values and you use those values to make decisions. Should I do this or should I do that exactly the same as every other value in your life you are raised and you learn and you develop your own values to help you make decisions.
And that's all the manifesto is. So it's not a failing of the manifesto that that's happened. Well, no, that's not true. It probably is a failing, but it's a failing because it had high expectations that people could value these things, quote universally and people don't.
They'll accept that as being a failing. I don't know what you do about it though, but there is the other side to that too. So for example, testing right in the first pragmatic program we went overboard with testing because people were not testing so many projects being written with zero tests.
What tests there were tended to be if it outputs this string, then it's correct. So we went over the top with testing and I think combining that with the XP practices of testing and all the rest.
People got religion and you know, around about the mid 2000s, it got to the point where people were shaming other people if they didn't have enough tests. I never write code unless there's a test blah, blah, blah. Well, that just makes you an idiot.
Testing is a tool like every other tool and you use it when you need to use it. And if you think agile means I have to write tests, then you are by definition not being agile because the agile view would be let's see what happens if we don't write tests.
You know, it kind of reminds me of when I and I and a few other folks we started a team at Amazon and basically we were free to set up our own processes as we wished. So we were like, let's do scrum the right way.
And then we started with doing those points as you know points they shouldn't equal you know man hours or whatever mandates and they should be like abstract points and after a few weeks we were like, why are we doing this? It just doesn't make any sense.
We figured out that we're spending more time in figuring out the points than the value that the points were giving us.
It was pointless. Very good. Yeah, but that kind of ceremony. That's the kind of thing that I object to so much. I mean, there's nothing fundamentally wrong with scrum except people do it as opposed to using it as the basis for developing your own practices.
I don't care if you go out and you kill a cat and do care about that, but in principle, I don't care if you go and kill a cat and read its entrails and use that to tell you how to write plough for as long as every day, every month, every year, you're looking at your processes and improving them based on your experience.
Because if you do that, it doesn't matter where you start, you will eventually get to a better place. I think the thing is that we start to create these templates, then we start to quantify the output of these templates with metrics and then these metrics become the thing that we're driving towards rather than the actual improvement of the process that we were seeking.
I mean, you're familiar with Taylorism. Yes, Frederick Taylor. Yeah. I'm not. Oh, Frederick Taylor was the original time and motion man. He went around companies and he measured with a stopwatch what people were doing and how they were being more effective or not effective. It's like 150 years ago.
Yeah. And he then used that to produce metrics standards against which people had to work. And it basically heralded in the kind of more assembly line based way of looking at working where people were became units of production and resources and not people.
Obviously, it worked, but it had limits. And that's when eventually the Japanese worked out as is a better way doing this. And we started to see Toyota come up with their techniques for managing production lines and stuff that worked better.
But yeah, you're absolutely right. Because what happens is if I can't put it in a spreadsheet, it doesn't exist if I'm a middle manager. So give me your metrics. Maybe that's a product. Forget about the podcasting.
What are products that just generates random number metrics that teams can give their managers and make up name make up names for the metrics. So no one actually knows what they are.
If you were part of the company, we might as well have done it, but we have to feed our families.
Oh, all right. No, we're running short on time and we just want to ask one final question.
So they view are very well known for the book. I mean, you've done other things too, but this is the thing that is like the main thing that you are known for.
It's been like quarter century. This was written. So how did that define your identity or do it? How much of it is a part of your identity?
Oh, that's a dangerous question. So I don't think it defined my identity internally in that to a large extent, it was simply recounting what I was doing anyway.
So in that way, it didn't change what I did. It, however, it gave me the ability to interact in circles that I previously didn't have any access to.
So I got invited to speak to conferences and other speakers would talk to me and, you know, we could have conversations and everything else.
So in that way, it dramatically broadened my access to cool new things and to people's opinions and to people's ideas.
At the same time, we, we called our business, the pragmatic programmers, after the book. And we called it the pragmatic bookshelf.
And my pretty much universal handle is priceless. So I guess you could say in that way, it is very much part of our identity.
I don't know how significant that is. It certainly doesn't hurt. I know that some people have done stuff like, well, say the manifesto and have then effectively pivoted what they did to be based on that and built in some cases, ridiculously successful companies based on the manifesto stuff.
And that was not of interest to me. It probably should have been because I probably worth considerably more than I am now.
But that wasn't what I wanted to do because it didn't seem to be that exciting, just like taking what we already had and rehashing it.
So in that way, I would say it probably has affected my density less than it has other people. I'm sure it has, but it's not a motivator in that way, not a driver.
Yeah, I think one of the things like this can affect identity, right? Other things that you decided not to do because you are the pragmatic Dave.
Oh, okay. Yeah, there is actually a really interesting little side effect. And that is whenever I'm doing something like writing code that's public back in my mind, I have to make sure I actually do it the way I said, said we should do it.
So yeah, it does affect me that way. It keeps me honest, let's put it that way. Yeah, makes a lot of sense.
So Dave, it has been a great pleasure to have you on the podcast. When you said yes, I think it took actually I had to work through your assistant or your marketing manager.
It probably took a couple of months before actually we got a response and oh, sorry about that. No, I mean, it's all good. Like we don't get the response from most people at all. So we were super excited.
This one was I told you, yeah, this is not like never coming back. So yeah, might as well ask you. No, I mean, honestly, I enjoy this kind of thing. I always learn because I always find that I'll come up with something that I haven't quite thought of that way before.
And people ask questions that just make me think about things and it's enjoyable. And you're nice people and it's nice to talk to nice people.
It was amazing. Yeah, thanks for coming.