00:00:09okay I guess we’ll start thanks very
00:00:13much for deciding to be in this talk
00:00:16rather than something else I’m Goku and
00:00:19what I’ll be talking about is how do
00:00:22service deployments impact how we design
00:00:27and deploy our systems I think one of
00:00:30the really interesting things that is
00:00:32happening at the moment is this whole
00:00:36buzzword is exploding and lots of people
00:00:40are approaching it from a technical
00:00:41perspective where is it
00:00:44stateless is it functions is it platform
00:00:47as a service and as an industry we like
00:00:49dealing with technical stuff that’s what
00:00:51we do tend to do but I think that the
00:00:56whole serverless thing whatever you
00:00:58decide to call it is much much much more
00:01:00important to look from a financial
00:01:03perspective so the way we are
00:01:07approaching things in terms of designing
00:01:10and deploying applications things we
00:01:12take for granted today I have been built
00:01:15based on our experiences in the last 20
00:01:1930 years deploying systems and many of
00:01:24the constraints that exists in the last
00:01:2820 30 years no longer applied to
00:01:31deploying with a SS lambda which means
00:01:34that many things that we now take for
00:01:37granted as best practices just are
00:01:39solutions for constraints that no longer
00:01:41apply and that was really really
00:01:44interesting for me kind of when we
00:01:46started getting our head around lambda
00:01:48deployments and that’s kind of what I
00:01:49want to talk about what are the things
00:01:51that we now think are best practices but
00:01:54are actually just solutions for problems
00:01:56that are no longer applicable and kind
00:01:59of in terms of what I’m gonna talk about
00:02:03just as an example I develop a
00:02:08collaboration up that helps people do
00:02:12mind maps online and in February 2016
00:02:15we started migrating from Heroku to a SS
00:02:18lambda it
00:02:19took us about one year to move
00:02:21everything because we did gradually
00:02:23during that year we increased them now
00:02:28kind of decreased the hosting costs by
00:02:30about 1/2 and while at the same time
00:02:34adding a bunch of new services and our
00:02:36number of active users increased by 50%
00:02:38in the same period so kind of all in all
00:02:41my estimate is that we saved around 66%
00:02:45or 2/3 on our hosting costs and that’s
00:02:49really really interesting if you look at
00:02:51it from a perspective running a small
00:02:52business
00:02:54so after kind of I started publishing on
00:02:58this a guy called Robert Charlie got in
00:03:00touch with me he wanted to do a
00:03:01scientific paper on this and together we
00:03:04wrote a proper scientific paper Martin
00:03:06talked about science papers yesterday so
00:03:08you know all my professors at the
00:03:10university of finally gonna be proud of
00:03:12me and know that all the alcohol kind of
00:03:14was worth something at the end and you
00:03:17can kind of download this it’s it’s
00:03:19called the kind of economic an
00:03:22architectural impact of cephalus is
00:03:24horrible but that’s okay so you can get
00:03:26a lot more on the numbers that I talked
00:03:27about there but I realized as we were
00:03:29doing research for this that actually
00:03:31our results were not even that
00:03:33interesting because we talked to people
00:03:36that saved something like 99% on their
00:03:39hosting costs by moving away from other
00:03:42platforms to – lambda’
00:03:45Heroku is reasonably cost efficient
00:03:48anyway and moving or moving from older
00:03:51generations of cloud hosting or moving
00:03:53from on-premise hosting to lambda has an
00:03:55even bigger potential to kind of do
00:03:57stuff so the key thing there I I think
00:04:01that is really important to consider is
00:04:03kind of the way lambda is priced and in
00:04:06my experience is mostly with AWS lambda
00:04:08although pretty much all the other cloud
00:04:11providers are copying features and
00:04:13models now so the way a double is lambda
00:04:15is priced kind of fundamentally changes
00:04:17the incentives for deployments
00:04:20fundamentally changes the incentives for
00:04:22good architecture and I think Martin’s
00:04:24talked about yesterday about engineering
00:04:26and how that’s working within
00:04:27constraints
00:04:28I think cost is one of the major
00:04:31constraints we have to work in
00:04:33and the pricing model kind of
00:04:37fundamentally changes really important
00:04:38so kind of in terms of the pricing lamda
00:04:41prices stuff per request and per 100
00:04:46millisecond increments in a processor in
00:04:50a virtual memory pack so these two
00:04:53things are really really important to
00:04:55consider because they change how we pay
00:05:00for for kind of what we’re using and
00:05:03they start charging things not in terms
00:05:07of reserved capacity but in terms of
00:05:10actual usage on a hundred millisecond
00:05:12increments so it doesn’t matter whether
00:05:15you have five VMs 500 VMs if you run
00:05:18three boxes or 5000 boxes to process
00:05:22something all that matters is how many
00:05:23requests came in and how long they took
00:05:27to execute under what memory conditions
00:05:29so this whole buzzword of services is is
00:05:33horrible because of course there are
00:05:34servers out there and things like that
00:05:36but as somebody came up with a real nice
00:05:39definition on Twitter the other day is
00:05:40saying that kind of that the thing is
00:05:42serverless
00:05:43if you’re only paying for actual usage
00:05:45if you’re not paying for what you have
00:05:48to plan as a reserved capacity so kind
00:05:51of historically that’s not been like
00:05:53that historically good architecture
00:05:56optimized for reserved resources and I
00:06:01you know 20 years ago in my previous
00:06:04life I worked on trading platforms that
00:06:06were deployed on immortal Hardware I was
00:06:09never supposed to die the storage cost
00:06:11more than my house the processors were
00:06:14insanely expensive and everything was
00:06:16duplicated replicated replicated because
00:06:20it’s never ever ever supposed to die but
00:06:23once you have a machine like that you
00:06:26optimize for using what you’ve bought
00:06:27you kind of you bundle stuff onto it you
00:06:30put everything you can to run there and
00:06:33you’re very very careful not to exceed
00:06:35the capacity of that because if you do
00:06:37then you know adding a couple of more
00:06:39processors or adding a bit more storage
00:06:41requires
00:06:42an insane amount of cost so then 2006
00:06:47Amazon kind of came out with the idea
00:06:50that you can get a virtual machine
00:06:51running in about five or ten minutes
00:06:53which was completely insane at the time
00:06:57I worked for a big telecoms provider
00:06:58where it took them nine months to
00:07:01provision a virtual machine internally
00:07:03and you know now comes Amazon you can
00:07:07get the VM for about ten minutes and and
00:07:09that there was amazing but kind of it
00:07:11did not fundamentally change how we are
00:07:13thinking about deployments because you
00:07:16got five virtual machines you’re paying
00:07:18for five virtual machines so you’re
00:07:19gonna bundle everything into those five
00:07:21virtual machines and we teach people
00:07:24good software design like decoupling
00:07:26isolation and and all those brilliant
00:07:29software design practices and then
00:07:31because you have five virtual machines
00:07:33then you put your payment servers and
00:07:34your log and your monitoring system all
00:07:38on the five virtual machines where they
00:07:40start interacting with each other and we
00:07:43had this problem we’re kind of deploying
00:07:45lots of different payment services to
00:07:46reserve VMs one of the VMS did not clean
00:07:50up the temp space correctly now one of
00:07:54the payment services did not clean up
00:07:56after itself and all the time it filled
00:07:58up the temp space Linux starts really
00:08:00really misbehaving when you fill up the
00:08:01temp space so that kind of VM started
00:08:04going crazy all the payment services
00:08:06that machine went down although they
00:08:08were designed to be decoupled and
00:08:09isolated and everything and that when
00:08:11that machine went down there was a
00:08:12cascade all you know the remaining four
00:08:15machines started filling up that time
00:08:17space very very quickly and everything
00:08:19kind of imploded
00:08:20so um although you know he teach people
00:08:23about good design as decoupling we end
00:08:26up deploying stuff to save cash because
00:08:28we need to reserve capacity and the next
00:08:31generation of cloud deployments Google
00:08:34App Engine Heroku and things like that
00:08:35they moved a lot of the responsibilities
00:08:39over to the cloud providers like
00:08:41provisioning like monitoring and things
00:08:44like that but the whole thing remained
00:08:46you were paying for dinos on Heroku for
00:08:48example we our app was deployed on
00:08:50Heroku we
00:08:51we have about 20 or 30 different
00:08:53exporters in two different formats and
00:08:56if I’m going to run a primary and a
00:08:58failover for each of these exporters on
00:09:00separate isolated VMs plus for some
00:09:03exporters I need a lot more capacity and
00:09:05for others for some one in five or six
00:09:06VMs that means that for 30 exporters I
00:09:11need a hundred VMs to run on if I really
00:09:13want to make it reliable and isolated
00:09:17but some of those exporters like
00:09:19markdown I wrote for myself nobody else
00:09:21uses that some of those exporters like
00:09:25PDF I use it all the time so you know
00:09:27I’ll put the market on exporter on the
00:09:29same block of machines as the PDF I’m
00:09:32not going to create a separate block of
00:09:33machines to save money and you know of
00:09:35course we made a stupid mistake and some
00:09:38of those things started interfering with
00:09:39each other because they were on the same
00:09:40machine now technically we have a
00:09:43solution for this and we had a solution
00:09:44for this it’s containers it’s isolation
00:09:46it’s things like that but as a you know
00:09:48average company it’s very very difficult
00:09:50to dedicate the resources to manage
00:09:54everything there so what lambda does
00:09:56Islam that kind of provides that as a
00:09:57service so we start moving away from
00:10:00reserved capacity to utilize capacity it
00:10:04doesn’t matter how many VMs you need
00:10:06Amazon is going to scale this and
00:10:08recharge you for how much you’ve used
00:10:09and that’s kind of the request based
00:10:11pricing that’s really really interesting
00:10:13so what that means is that there’s no
00:10:15more financial incentive for me to put
00:10:17the markdown in the PDF exporter on the
00:10:19same VM even worse because they have
00:10:24different memory needs markdown doesn’t
00:10:26need any memory PDF needs a lot because
00:10:28it’s using go script and go script is a
00:10:30memory hog putting those two things on
00:10:32the same VM I will end up paying for a
00:10:34markdown for a high memory watermark
00:10:36that’s a lot more than I need if I
00:10:38separate them so the financial
00:10:39incentives here are actually for people
00:10:42to unbundle the apps in all these small
00:10:44isolated modules it’s really really
00:10:46interesting because said historically it
00:10:48was completely different so one of the
00:10:51things that I think kind of people
00:10:54should think about more and it’s not
00:10:56microservices it’s not kind of functions
00:10:58you can you can buzz for this any way
00:10:59you want but I think what we started
00:11:02thinking about is really
00:11:03decoupling different tasks that have
00:11:06different memory constraints different
00:11:07CPU constraints different deployment
00:11:10needs and thinking about well you know
00:11:13if it doesn’t matter how many VMs this
00:11:15runs on I don’t care about it being
00:11:18separately deployed and that’s very very
00:11:20liberating and it’s completely opposite
00:11:23of what we’ve been taught to do a lot of
00:11:26kind of many practices like Bluegreen
00:11:28deployments many practices like kind of
00:11:30concentrated monitoring and things like
00:11:31that they did no longer that problematic
00:11:35and that opens up some really
00:11:38interesting possibilities for example
00:11:40moving from Heroku to lambda we’ve kind
00:11:42of deleted a bunch of code that was –
00:11:45dealing with kind of you know figuring
00:11:47out what’s going on on a particular VM
00:11:49figuring out how these things interact
00:11:50with do they interact with each other so
00:11:52a lot of this infrastructure code is
00:11:54gone so the next thing that kind of I
00:11:58think historically happened and no
00:11:59longer applies is generally good
00:12:01architecture because of the whole idea
00:12:03of you know reserved stuff optimized for
00:12:07failovers and what that meant was really
00:12:11complex state management you wanted to
00:12:13have a warm failover machine or a couple
00:12:16of warm fill of machines that can take
00:12:18the load in you know very quickly if
00:12:21something fails so data needs to be
00:12:23replicated cache needs to be replicated
00:12:25data needs to be synchronized across
00:12:28replicated machines then this kind of
00:12:31cache invalidation policies people need
00:12:32to care about this there’s a lot of
00:12:34really complex code to do to make this
00:12:38work really really well but from a
00:12:41perspective of paying per request kind
00:12:45of failover machines are not doing
00:12:47anything so kind of unless the receiving
00:12:48request you’re not paying for them and
00:12:50then the big question becomes do you
00:12:52actually need to plan for failover like
00:12:54that and another really interesting
00:12:56constraint that I think Canova’s is
00:12:58changing with the whole lambda
00:12:59architecture is kind of the time to
00:13:02recover that was historically a big
00:13:06driver for a lot of architectural
00:13:07practices is no longer that important
00:13:10and that opens up some really
00:13:12interesting possibilities what I think
00:13:13is becoming a lot more important is the
00:13:15time to start
00:13:17so when we talk about the time to start
00:13:20I’ll just show you something very
00:13:22quickly let’s see if the gods of the
00:13:23Internet help us here so ok so if I do
00:13:37[Music]
00:13:39and I do constable let’s be nice
00:14:17so this is kind of a a web api that on
00:14:21any request to slash hello will reply
00:14:23with hi there it’s not that’s kind of
00:14:27complicated
00:14:28and if I’ve not made a stupid mistake we
00:14:30should be able to do something like so
00:14:43I’m using an open-source tool for
00:14:46deployments here that actually we’ve
00:14:48open sourced while building it for
00:14:51mindmap the first one that we created
00:14:53was something like 30 lines of
00:14:55JavaScript code and about 250 lines of
00:14:57shell scripts to deploy and then I
00:14:59realized that kind of the risk is no
00:15:01longer in the code the risk is in the
00:15:02deployment so we wanted to have properly
00:15:05unit tested deployment and configuration
00:15:06and and we kind of just packaged up all
00:15:09the scripts so this thing is now on on a
00:15:11URL on Amazon and if I grab this URL so
00:15:20I’ve created a slash hello so if I do
00:15:22curl hello I get to hi there
00:15:27so now this is a kind of thing that’s
00:15:29running on one VM but what’s really
00:15:32interesting about this is that if all of
00:15:36you start attacking this thing now it
00:15:38will perfectly handle the load if
00:15:39nobody’s using it they don’t have to pay
00:15:41for anything if we have a massive
00:15:44massive spike now but then kind of you
00:15:46know the spike goes out that’s perfectly
00:15:47fine as well it will scale up and scale
00:15:49down plus with this I get a bunch of
00:15:52things automatically like monitoring
00:15:54logging security provisioning all the
00:15:58stuff that kind of people want to deal
00:15:59with the operations so I’ve created this
00:16:01thing called go to Copenhagen so if I go
00:16:04to my lambdas now
00:16:12come on internet okay so we have a knob
00:16:20that’s not the right account
00:16:41goodgoodgood
00:16:42so let’s see Amazon four one three six
00:16:47so for now you’ve seen my code come on
00:16:54come on so if I go to lambda now I have
00:17:07no idea what they’ve changed here so
00:17:09this is the okay III you can see how
00:17:17often I use this anomaly use the command
00:17:19line D so what I want you to say is kind
00:17:23of with with this you get a bunch of
00:17:24operations things for free immediately
00:17:26and why is this suffering bustle and
00:17:29things that doesn’t matter
00:17:30let’s functions functions functions
00:17:35functions page to go to Copenhagen okay
00:17:42so here it is and then I have kind of
00:17:44lots of stuff around monitoring I have
00:17:47my kind of errors throttles I have logs
00:17:51in cloud watch I have pretty much
00:17:52everything I need from an Operations
00:17:54perspective already provided to me so
00:17:57there’s this whole buzzword kind of that
00:18:00lambda is moving from DevOps to no ops
00:18:02and Simon worldly who’s a researcher in
00:18:05the UK said kind of your company’s not
00:18:07done DevOps yet just don’t bother skip a
00:18:09whole generation of things and move to
00:18:13something like this and it’s easy
00:18:14fair enough it’s not completely killing
00:18:16DevOps but now I said you know you have
00:18:18all the operations stuff already I have
00:18:20a log I have kind of when it started
00:18:22when it ended how much memory it used
00:18:24and this is amazingly good for
00:18:27optimizing stuff if you look at kind of
00:18:30the typical architecture and and and
00:18:32stuff like that we normally had figuring
00:18:35out how much money a particular function
00:18:38cost you to run was almost impossible
00:18:39people are optimizing based on a gut
00:18:42feel they were optimizing whole
00:18:44applications now I know exactly for this
00:18:46particular task how much money I’m
00:18:47spending so I can decide do I need to
00:18:50kind of invest in optimizing that or not
00:18:52so that’s really really really
00:18:53interesting and kind of in terms of time
00:18:55to start here’s some here’s some
00:18:57empirical numbers that I got these are
00:19:00not confirmed by the AWS a Tablas
00:19:02doesn’t publish any numbers about this
00:19:03but these are kind of my numbers that we
00:19:06got generally so for a completely new
00:19:09instance of a deployed up so if you
00:19:13start hitting this stuff now and this
00:19:14instance that it’s running their content
00:19:16load with the JavaScript
00:19:18this lady takes less than one second to
00:19:21get the new instance if you want to do a
00:19:23new deployment a new version like I’ve
00:19:25just done it takes about three or four
00:19:26seconds to kind of set everything up so
00:19:29that’s the infrastructure stuff now
00:19:32what’s left is how long our app takes to
00:19:36connect to the database load up the
00:19:38cache load up the data and things that’s
00:19:39I think kind of unlike the previous
00:19:42architectures where we were optimizing
00:19:44for quick failover what we need to start
00:19:45thinking about is optimizing for quick
00:19:47start lazy loading everything kind of
00:19:50making sure that we don’t add a lot of
00:19:53this stuff because if we can then does
00:19:54this stop just kind of starts and dies
00:19:56on its own now there’s a whole buzzword
00:19:58here where people are talking about how
00:20:00we need to optimize we need to write
00:20:03this stuff in a stateless way because
00:20:08there’s a big confusion of buzzwords and
00:20:12some people talk about services function
00:20:15as a service than in a functional
00:20:16programming and functional programming
00:20:17is stateless and this you know people
00:20:19throw buzzwords around lambdas are not
00:20:22stateless don’t have to anybody tell you
00:20:23that lambdas are a stateful container
00:20:27just having no idea how many containers
00:20:29you’re running once the container starts
00:20:32you have no control over when is it
00:20:33going to stop and whether the next
00:20:35request from the same user is going to
00:20:37hit the same container or a completely
00:20:38different container so rather than
00:20:40developing for stateless what I think we
00:20:43should be thinking about this developing
00:20:44kind of for sure nothing designing for
00:20:47sure nothing when the VM load is
00:20:49perfectly fine to start caching things
00:20:51in that VM that are not user specific
00:20:53that’s how we save a lot of time on
00:20:56processing requests that’s how we save a
00:20:57lot of cash because you know that there
00:21:00will be a VM running it’s just that you
00:21:02mustn’t ever catch anything that is
00:21:04specific to that particular request the
00:21:06state of a user session is not there the
00:21:09state of a user session needs to be
00:21:10somewhere else and we’ll talk about
00:21:13quite an interesting option where to
00:21:15push the user state later but kind of
00:21:19it’s it’s not stateless people that have
00:21:22never kind of deployed any of this talk
00:21:24about lambdas being stateless they’re
00:21:26not and and I think you lose a lot of
00:21:28money by treating them like stateless so
00:21:30the next thing that I think kind of
00:21:32historically happened is it was really
00:21:35really difficult to replicate production
00:21:37it was you know people had a production
00:21:41system with expensive storage or even if
00:21:44it’s on the cloud with good VMs and then
00:21:46that cost a lot so you wouldn’t want to
00:21:49pay the same amount for your staging you
00:21:51didn’t want to pay the same amount for
00:21:52the testing so typically kind of
00:21:55performance tests unless you are Lmax
00:21:57would run on a kind of a smaller copy of
00:22:01data on a smaller copy of the production
00:22:04and things like that they were not
00:22:05really relevant they were not reliable
00:22:07and generally that made it really really
00:22:09difficult to claim anything about
00:22:12production usage and production
00:22:14performance based on the testing system
00:22:16now kind of I have a couple of friends
00:22:22from Australia and and they they claim
00:22:24that I don’t know if this is true or not
00:22:26the time that kind of Facebook is
00:22:27continuously broken in Australia because
00:22:30Facebook is really good with kind of
00:22:32running business experiments on all
00:22:33their users and kind of Australia’s the
00:22:36market that’s big enough to be
00:22:38statistically significant it’s English
00:22:40so they don’t have to waste a lot of
00:22:41time preparing software for that but
00:22:43Facebook doesn’t really care too much
00:22:45about Australians they’re laid-back
00:22:46people anyway so they can do experiments
00:22:49in Australia and then when they figure
00:22:51what works what doesn’t work they can
00:22:52translate that to the US now when you
00:22:55think about something like that that’s
00:22:57incredibly useful but up until five
00:23:00years ago or up until three years ago
00:23:01that was available to you if your Google
00:23:04or Facebook running experiments like
00:23:07that is you feel expensive you need a
00:23:09complete copy of production for
00:23:11something like that and you need the
00:23:14relevant copy of production you need to
00:23:16synchronize the data between these
00:23:17things what one of them kind of best
00:23:19examples of how important that is is a
00:23:22story called 40 shades of blue they’re
00:23:25absolutely loved because it’s one of
00:23:27those rare things where you can read
00:23:29both sides of the story online forty
00:23:32shades of blue led to kind of that the
00:23:34head of design at Google wanted to
00:23:37change the color of the links on the
00:23:39homepage for ads and the developers
00:23:42challenged him a bit
00:23:44and he wanted to change it to a
00:23:46particular blue collar overnight they
00:23:48ran 40 different colors of blue and
00:23:51measured how much people are clicking on
00:23:54that and then as a result of that
00:23:56depending whose side you read this guy
00:23:58either quitter was fired because the
00:24:02difference was something like 250
00:24:03million dollars between the current
00:24:06color and his color if you expand it to
00:24:08a whole whole year and 100% of the users
00:24:10so kind of things like that are
00:24:14incredibly important to test but for
00:24:18most people out there it was almost
00:24:20impossible to do it because production
00:24:22copy cost so much now when we started
00:24:24doing lambda I had this brilliant best
00:24:26idea in the world and I wanted to
00:24:29integrate wiki data graph of knowledge
00:24:33with our app so that when you press a
00:24:35question mark it automatically opens up
00:24:37related terms it was going to be amazing
00:24:38it was going to be brilliant it’s going
00:24:40to be fantastic and and my business
00:24:42partner said no that’s a shitty idea
00:24:43nobody’s going to use that let’s not
00:24:47waste time doing that so you know I kind
00:24:49of generally don’t like to ask for
00:24:52permission like to ask for forgiveness
00:24:53so I decided I’m going to do that anyway
00:24:55and he said I know you’re going to do it
00:24:56anyway do not touch the
00:24:58production code because if you put this
00:25:00in we’re going to have people you know
00:25:02screaming that we can’t support the
00:25:04right performance we’re gonna be bottom
00:25:05like here we’re gonna and I said okay
00:25:07and then I realized kind of with lambdas
00:25:10you don’t pay for stuff if you people
00:25:12don’t use them so I kind of spent two
00:25:15days knocked up a quick lambda version
00:25:18that was extended with this deployed a
00:25:20completely separate copy of our
00:25:23production and sent 20% of our users
00:25:26there and you know lo and behold I
00:25:29proved that it was a idea nobody
00:25:31wants to use it so we deleted that code
00:25:34with our disturbing uh production so but
00:25:39this is this is some you know something
00:25:41that traditionally we’d get into a fight
00:25:43who’s right who’s wrong guy would do it
00:25:45anyway and then that code would stay in
00:25:46the production and kind of you know cost
00:25:48us more to maintain it was amazing
00:25:50because we could run a quick experiment
00:25:51so because we’re paying for requests
00:25:53we’re not paying for reserve capacity it
00:25:56was exactly the same amount of money to
00:25:57send 80% of users to one version 20% to
00:26:00another or to send everybody to the same
00:26:03version or to have a version for
00:26:04Copenhagen or to have a version for
00:26:06go-to or to have a version for anything
00:26:08you want and actually kind of what this
00:26:10starts getting us to think about is
00:26:12moving away from thinking about
00:26:13production to thinking about multiple
00:26:15versions and multi versioning in lambda
00:26:17is incredibly well done I you know for
00:26:22my scenes like everybody else have done
00:26:24an Orion framework a logging framework
00:26:25and a data multi version in framework
00:26:27everybody’s done that when they young
00:26:28otherwise you don’t you know you don’t
00:26:30get to call yourself a developer and
00:26:32then you end up kind of suffering
00:26:34through maintaining that for 10
00:26:36years but that’s okay so a data multi
00:26:41versioning request multi versioning
00:26:42service multi versioning is really
00:26:43really difficult to do well at scale and
00:26:45Amazon probably have done it for their
00:26:47own needs and they’ve just exposed a T
00:26:49lambda so every lambda function gets a
00:26:51numerical version every time you deploy
00:26:53it and you can direct the calls for a
00:26:58particular function either to the last
00:27:01non-deployed version or to a particular
00:27:04numerical deployment or you can assign
00:27:06aliases to a numerical deployment like
00:27:08production testing staging so when we
00:27:10started doing this we started really
00:27:12kind of doing oh I have a production
00:27:14version a testing version and staging
00:27:15version and then we realized why I’ll
00:27:18just save a version to test this feature
00:27:19oh I’ll have a version to kind of you
00:27:21know when we’re doing something
00:27:22experimental and 5% of our users really
00:27:24really need this feature but it’s not
00:27:27ready for everybody else we can deploy a
00:27:29version for them and give it to them and
00:27:30test it on them early and it’s multi
00:27:33version is incredibly well done so I
00:27:35think this is a really interesting thing
00:27:36to start thinking about kind of how do
00:27:38we bundle our tasks because it has a
00:27:40massive impact on how we design how we
00:27:43deploy if we are going to design for a
00:27:45multi version universe where you know at
00:27:48the same time several versions of a
00:27:50wrapper running and different things are
00:27:51communicating with even I think the
00:27:52whole domain driven design concept of
00:27:54aggregates becomes incredibly more
00:27:56important because we want to make sure
00:27:59that kind of you know all the data for a
00:28:01particular version of this object
00:28:02travels together and because we are kind
00:28:06of don’t want to clog the communication
00:28:08things that we want to make sure
00:28:09aggregates are actually relatively
00:28:10minimal so that we can push them around
00:28:13and we can load them quickly but the
00:28:17movie to land has gotten us to thinking
00:28:20a lot more about what are our actual
00:28:21aggregates where the aggregate
00:28:23boundaries what needs to be in a
00:28:26particular version consistent with
00:28:28itself what can just kind of be
00:28:29different versions and and be okay so I
00:28:32think that’s a completely interesting
00:28:34kind of phenomenon that are not really
00:28:35seen in my code before that and because
00:28:38we’ve designed for multi versioning now
00:28:40up front as we’re migrating we can do
00:28:42lots of crazy things and the stuff that
00:28:44was the traditional available to you
00:28:46know companies that make billions and
00:28:48billions of dollars we can do now and
00:28:50we’re a two-person team and that that’s
00:28:53I think amazing as a kind of for what we
00:28:56can do from the platform so the next
00:29:00really interesting thing driven by the
00:29:03pricing model of AWS lambda is that
00:29:06different services charge for different
00:29:08things so you can save quite a lot of
00:29:10money by playing arbitrage available yes
00:29:13against AWS that’s amazing
00:29:15and remember kind of lambda is as a
00:29:18processing service charges for the
00:29:20number of requests in time so if you can
00:29:23delegate work from lambda to stuff that
00:29:26does not charge for the number of
00:29:28requests in time you can save a lot of
00:29:32cash for example kind of API gateway
00:29:35charges for the bytes being transferred
00:29:38the number of requests at the same time
00:29:41kind of s3 that’s the storage system
00:29:43just charges for transfer it doesn’t
00:29:45care about the number of requests so one
00:29:49example of that is that we own Heroku
00:29:52and before we started really thinking
00:29:53about this we would let people upload
00:29:58files for export to a server where the
00:30:02server would communicate with the
00:30:04storage and then the server would save
00:30:07stuff through storage it with rondo
00:30:08converter it would kind of upload it
00:30:10back from the storage to the user what
00:30:12that means is that during that whole
00:30:13time the server is busy we’re paying for
00:30:16the server now if you’re uploading a
00:30:19hundred megabyte file the transfer from
00:30:21you to
00:30:22Amazon and the transfer for Amazon to
00:30:25you is actually the most amount of time
00:30:26the conversion time is relatively quick
00:30:28so we’re paying kind of for this
00:30:30operation for a long time we’re because
00:30:33there’s three only pays s3 only charges
00:30:35for transfer if we can get people to
00:30:36upload directly to s3 and download
00:30:38directly from s3 we have reduced our
00:30:42server costs by a significant amount so
00:30:45some other service that are interesting
00:30:47to consider like kognito the Amazon
00:30:49authentication and session service only
00:30:50charges for the number of users it
00:30:53doesn’t charge for the number of
00:30:54requests those users make it doesn’t
00:30:55charge for the capacity of the sessions
00:30:58it only charges for the number of users
00:31:00so moving session state into kognito
00:31:02it’s a really really interesting way of
00:31:05kind of playing arbitrage with Amazon
00:31:06versus Amazon so kind of as a kind of an
00:31:11example I’ll show you later there’s also
00:31:12this thing called the IOT gateway that
00:31:14we abuse massively IOT gateway is
00:31:17designed to get low-power devices to
00:31:19talk to each other but we’ve built kind
00:31:21of real-time collaboration directly
00:31:23through we’re building real-time
00:31:25collaboration directly through IOT
00:31:26gateway which kind of because it only
00:31:28charges for the number of messages it
00:31:30doesn’t charge for processing time it
00:31:32doesn’t charge for data transfer again
00:31:34you can arbitrage things nicely so kind
00:31:36of we started moving a lot more for
00:31:40thinking about applications and and kind
00:31:43of managed apps like Heroku like Google
00:31:46App Engine to really the glue between
00:31:47different platform services for me
00:31:50lambda is mostly about what’s missing
00:31:53from amazon’s platform and how do i kind
00:31:55of glue those things things together and
00:31:57what’s the kind of what’s the real
00:31:59business value of my code because it’s
00:32:02unlikely that developing a queue
00:32:04interface system is where i can provide
00:32:06the most value i can provide value
00:32:08developing kind of the small bits of
00:32:10processing those queue messages and
00:32:12that’s really really interesting so kind
00:32:14of just as an example that this is kind
00:32:17of how to get people to use s3 directly
00:32:21from a browser so on a server we have
00:32:23something like this the s3 is the Amazon
00:32:26API SDK so you get a request for upload
00:32:30file you populated with you know where
00:32:32it needs to go you limit the file size
00:32:34you
00:32:35kind of provides security stuff and then
00:32:37you get a signature and then returned
00:32:39that to the browser that takes you know
00:32:4120 milliseconds then the browser spends
00:32:44ten minutes uploading the big file to s3
00:32:46where you’re just paying for transfer
00:32:48then kind of this thing goes very
00:32:50quickly you go back and people download
00:32:53it directly from s3 from a signed URL so
00:32:56amazon being amazon of course there’s
00:32:58this five different ways of authorizing
00:33:00requests
00:33:01there’s signed urls there’s the cig v4
00:33:05that they do for kind of approaches this
00:33:06is giving people access to kognito so
00:33:09this lots and lots of ways how you can
00:33:11get people to access directly one of
00:33:13these front-end services without going
00:33:16through a traditional server so kind of
00:33:19in to really benefit from that
00:33:21financially I think what we started
00:33:23thinking about a lot more is give the
00:33:25platform the roles that were
00:33:26traditionally associate with the server
00:33:28process things like being the Gateway
00:33:31keeper things like being the
00:33:32orchestrator things like keeping kind of
00:33:34sensation storage if you push that away
00:33:38from the lambda that’s kind of the
00:33:41processor to the other parts of the
00:33:43platform you can save a lot of cash now
00:33:46here’s why I think this is so insane in
00:33:54September our app had something like
00:33:57400,000 active users it’s it’s not you
00:34:01know Google but it’s not Mickey Mouse as
00:34:03well and just so that I’m not cheating
00:34:06this is V live Amazon page so if I go to
00:34:14my billing dashboard and I look at my
00:34:19costs for where is it the September bill
00:34:24bill details September so for four
00:34:29hundred thousand active users in
00:34:30September we have paid 53 cents
00:34:38for lambda now beat that with your
00:34:41hosting costs and we you know they of
00:34:45course there’s there’s some other
00:34:47services like we paid four dollars for
00:34:49data transfer and then we paid something
00:34:52for API gateway and something for dynamo
00:34:54and things like that but all in all that
00:34:56the bill was a hundred bucks for a you
00:34:59know four hundred thousand active users
00:35:01that are kind of collaborating in real
00:35:03time now this is insane
00:35:05completely insane if you look at kind of
00:35:07stuff equivalent stuff I was doing ten
00:35:11years ago that this is completely
00:35:13completely insane and you know add up
00:35:17all the multi versioning and everything
00:35:20else they provide for almost not free
00:35:23but included in the price that’s that’s
00:35:26why I think this is gorgeous and and and
00:35:29you know fantastic in so many ways so
00:35:30kind of the another thing that started
00:35:35happening here as we started thinking
00:35:37about more and more of arbitrage in
00:35:39different services against each other we
00:35:40realized that kind of what engrained in
00:35:44my head for the last 30 years is do not
00:35:48trust users to talk to Becky and
00:35:50resources like users are not allowed to
00:35:53talk to storage directly users are never
00:35:57ever ever allowed to connect your
00:35:58database directly they have to go
00:36:01through a gatekeeper they have to go
00:36:03through a server because on the server
00:36:05we discard invalid requests we validate
00:36:08stuff we you know everything before the
00:36:10server we don’t trust everything after
00:36:12the server we trust that’s how we you
00:36:14know did the whole web logic thing
00:36:17evolved we have application servers we
00:36:19have storage we have clients and I think
00:36:23kind of especially if you start to use
00:36:25the platform on Amazon none of these
00:36:28things are actually physically back-end
00:36:30resources anymore s3 is available over
00:36:33HTTP dynamo that’s the databases
00:36:35available over HTTP the fact that we are
00:36:39not letting users talk to you directly
00:36:42doesn’t mean it’s not available if
00:36:44somebody kind of guesses the name
00:36:46it’s there Amazon is making it available
00:36:48and because of that kind of Amazon is
00:36:51actually implementing really really good
00:36:54request level authorization policies
00:36:56each single request going from lambda to
00:37:01a database is authorized because if they
00:37:03have no idea if your lambda is talking
00:37:05to the database if somebody else Islam
00:37:06days talking to derivative is and it’s
00:37:08not your database anywhere it’s their
00:37:09database and it’s kind of things like
00:37:14this are really really interesting from
00:37:15a perspective of thinking about well you
00:37:17know if it’s authorized per request
00:37:22what’s the damage in actually kind of
00:37:25authorizing clients to go there so
00:37:27Amazon gives you three or four different
00:37:29mechanisms for authorizing these
00:37:31requests including say including saying
00:37:33that this user is only allowed to write
00:37:35to this particular key in the database
00:37:37and only allowed to read from these keys
00:37:39in the database hierarchically or this
00:37:41user is only allowed to read from this
00:37:43part of the queue and post to these
00:37:45parts of the queue and the same
00:37:48authorization policies exactly the same
00:37:50apply as if you talk to from the client
00:37:52there then if you go through the server
00:37:54and then I realized well you know we’re
00:37:57just introducing latency by putting a
00:37:58server in the middle we’re just paying
00:38:00more and we started kind of using this
00:38:03like mad my brain still does not allow
00:38:07me to or let users connect to the
00:38:09database because I’ve been a server-side
00:38:12developer for 20 years and it’s just
00:38:14wrong
00:38:15but we’re letting people talk to the
00:38:18storage directly well I think people
00:38:19talk to the queues and and kind of
00:38:21things like so you know maybe two years
00:38:23from now I’ll I’ll come into the talk
00:38:24and say no no everybody’s you know
00:38:25talking to the database and all our data
00:38:27is stolen and it’s horrible or you know
00:38:30but generally if you think about it’s
00:38:32not your database it’s Amazon’s database
00:38:33and it’s available using HTTP so it’s
00:38:38not back-end it’s content or its
00:38:41middlewares of some kind so kind of we
00:38:44started moving away from kind of
00:38:46three-tier models to smart clients not
00:38:48not dumb terminal smart terminals where
00:38:51things connect directly so here’s a URL
00:38:55and this is a
00:38:58a prototype we’ve developed for a chat
00:39:00app that works on all browsers and works
00:39:04on all mobile devices and things like
00:39:06that and kind of it
00:39:09the source code is on github you can
00:39:11look it up afterwards so kind of connect
00:39:24to this from your mobile phone or
00:39:26something like that and then you can
00:39:27kind of log in as a guest what that
00:39:30means is that we are now getting a
00:39:32authorization ID from Cognito I can make
00:39:36this login with the username and
00:39:37password log in through Google login
00:39:39through Facebook log in through many
00:39:41many different ways do two-factor
00:39:42authentication for a conference kind of
00:39:44show it’s you know open and then once I
00:39:49have this I can actually use their API
00:39:51directly from a browser to talk to any
00:39:54resource I want that I’m allowed to talk
00:39:55to so in this case we are talking
00:39:57directly to if you cannot get into the
00:40:00server listing just pump it up we are
00:40:02talking to the IOT gateway the IOT
00:40:08gateway is designed to kind of exchange
00:40:09messages between low-power devices but
00:40:11it’s actually allowing you to have a
00:40:15WebSocket interface as well how cool is
00:40:17that so you can use WebSockets on demand
00:40:20managed that paid $5 per million
00:40:26messages so peanuts and they’re
00:40:30completely managed so you know we can
00:40:33get a million people connecting to this
00:40:35now or we can get five people connecting
00:40:38to this and it’s it’s all done
00:40:40operationally the source code is 30
00:40:42lines of code so it is just completely
00:40:45insane what we can do now with this
00:40:47stuff and how much it costs and I you
00:40:50know ten years ago I remember kind of
00:40:52evaluating lots of different push
00:40:53mechanisms where they were doing
00:40:55degrading of a flash they were doing
00:40:57long pole they were doing all these
00:40:59amazing things and I think the best
00:41:00thing we’ll choose they were asking for
00:41:02something like a hundred thousand quid a
00:41:04month I can get this now for five
00:41:08dollars on a million messages if
00:41:10nobody’s using it
00:41:11I don’t pay if people are using it and
00:41:14probably making money of it so you know
00:41:15perfectly fine to pay but there’s no
00:41:18upfront cost there’s no monthly
00:41:20maintenance cost this is insane
00:41:22it’s completely since I think kind of
00:41:24this thing changes how we approach what
00:41:27we reserve what we what we kind of do
00:41:30and I said you just format see a nice ah
00:41:36good good good good good
00:41:38so um so yeah I said that is you know
00:41:42from here you can just go to go to the
00:41:44Lincoln and you can get the the source
00:41:46code for this so kind of in that respect
00:41:50what I want to say is good engineering
00:41:55who the architecture is driven by
00:41:57constraints cost is one of the key
00:41:59constraints we have to deal with and
00:42:01deploying on lambda fundamentally
00:42:03changes the cost structure so lots of
00:42:06stuff that you know have evolved as best
00:42:08practices over the last 20 years kind of
00:42:12no longer apply and our challenge as a
00:42:15community over the next five 10 years is
00:42:16going to be to figure out what are you
00:42:19know just the shackles of the old world
00:42:21that we’re running with I mean in it you
00:42:23know I I can talk what’s wrong about
00:42:27lambda 4 you know days on end and this
00:42:29is not a silver bullet it doesn’t solve
00:42:31all the problems but I think it’s a
00:42:33really really interesting perspective if
00:42:35you can you know run a four hundred
00:42:38thousand uses and pay fifty three cents
00:42:40for the whole thing it’s just insane and
00:42:43I think kind of the financial incentives
00:42:45of that are going to get pretty much
00:42:47everything that can run in lambda to run
00:42:49in lambda over the next five years and
00:42:52that’s why you know although most people
00:42:55here I assume are not really deploying
00:42:57things in production for lambda yet this
00:42:59will comes to start investigating that
00:43:01and in particular running cheap
00:43:03experiments is amazing if you need to
00:43:06run a kind of cheap stupid experiment
00:43:08and you don’t know if it’s going to work
00:43:09out or not this is brilliant for that
00:43:10and then even if you do and kind of
00:43:12on-premise deployment later then you
00:43:14know what to integrate and and and what
00:43:16to throw away so kind of I think one of
00:43:21the key things that was you know a mind
00:43:24shift for us is
00:43:25is to start letting clients connect to
00:43:27back-end resources because there are no
00:43:28back-end resources anymore and what that
00:43:30means is that all of the sudden your app
00:43:32is not really running just on fifty or
00:43:36five or five hundred virtual machines
00:43:38it’s running on four hundred thousand
00:43:40client processes as well because you can
00:43:41push if you let people talk to Becky and
00:43:44resources you can push orchestration you
00:43:46can push state to the client and all the
00:43:51stuff that kind of is difficult to
00:43:52manage in the distributed architectures
00:43:54push it to the clients machine where the
00:43:56client is a single client having a
00:43:58single state which simplifies things
00:44:00significantly so instead of kind of that
00:44:03that’s why we pay so little for lambda
00:44:04our app does not run on the VMS what
00:44:07runs on the VMS is a glue between
00:44:08different back-end services our app
00:44:11actually runs on four hundred thousand
00:44:14client processes that we do not have to
00:44:15pay for and that’s kind of the really
00:44:19really interesting mind shift here so
00:44:21kind of as two URLs for for more info
00:44:25this first one is my blog where I post a
00:44:29lot about this stuff because I’m
00:44:30incredibly excited about it the second
00:44:33one is the open source tool for
00:44:34deployment that I’ve shown you that kind
00:44:36of simplify stuff if you do in
00:44:38JavaScript that’s pretty much it thank
00:44:41you very much I hope I kind of tickled
00:44:43your imagination at least of it
00:44:45and if anybody’s posted any questions I
00:44:47guess we can talk about that now do we
00:44:50have any questions where’s the I can
00:44:53read it loud okay lovely how do you feel
00:44:56about locking kind of that that’s a
00:45:00really interesting question and
00:45:02unlocking is is problematic on several
00:45:05levels lots of people talk about locking
00:45:08in terms of locking with libraries
00:45:10locking with code if you use Oracle then
00:45:12you know you use Oracle’s libraries with
00:45:14lambda because the platform calls you
00:45:16not the other way around you’re actually
00:45:18not locked into the lambda API at all
00:45:20there aren’t it’s it’s moving away from
00:45:22there would be trivial the big problem
00:45:25is you locked into the platform if you
00:45:27really want to get the benefits of
00:45:28lambda then you’re letting clients talk
00:45:29to storage directly I think lines talk
00:45:31to the database correctly and that’s
00:45:33where the locking happens now for us
00:45:36we’ve kind of we we’ve decide
00:45:38commercially that going really for
00:45:41Amazon and and using Amazon for
00:45:42everything is a good commercial decision
00:45:46it gives us the risk Obama’s are not
00:45:48working but in my experience kind of
00:45:49they’re pretty solid and they’re much
00:45:51much better doing the OP stand I can so
00:45:55I I know that there are some tools like
00:45:57the serverless framework that allow you
00:45:59to deploy to multiple clouds and do kind
00:46:02of this hybrid thing but I guess the big
00:46:04problem then like you know doing a
00:46:06database independent deployment is you
00:46:09get to use the the least common
00:46:10denominator of the whole thing and
00:46:11you’re never really using the platform
00:46:13what it is so kind of that’s a
00:46:16commercial decision that I guess
00:46:17everybody needs to make on their own but
00:46:19there’s definitely things like you know
00:46:20ports and adapters or hexagonal
00:46:22architecture and things that you can
00:46:24design stuff so if you do actually
00:46:26decide to move on to different cloud
00:46:27provider you know you will I I don’t
00:46:30I’ve never worked with a company where
00:46:32the whole investment in being able to
00:46:35move from the primary database paid off
00:46:37because they never moved away from the
00:46:39primary database so it’s a commercial
00:46:41decision there’s only two of us building
00:46:43this thing so I’d rather spend stuff
00:46:45delivering successful features than
00:46:47building kind of an abstract software
00:46:49system but you know if you have five
00:46:51kind of developers why not keep them
00:46:54busy so so the this disorder locked your
00:46:59kind of can I get Apple pay on your
00:47:01phone as well when you unlock it okay so
00:47:06we have how do you keep the state we
00:47:09decision so we tend to push a lot to the
00:47:11clients directly and and we tend to keep
00:47:13the state in either incognito or in the
00:47:15users browsers kognito does automatic
00:47:17synchronization across devices we don’t
00:47:21tend to use that a lot because our our
00:47:22state is typically kind of the document
00:47:24you’re working on and we don’t need to
00:47:28keep a lot of that but kognito is a
00:47:29pretty good way of kind of synchronizing
00:47:32stuff across devices using dynamo using
00:47:35something like that to keep kind of the
00:47:36state per user is also pretty good
00:47:38because you can configure dynamo to
00:47:41allow users to write only to a
00:47:43particular key so you ko or or a sub key
00:47:46so you can write only to your own state
00:47:48and read only from your own state for
00:47:50example that
00:47:51that would be a possibility mm-hmm if
00:47:58I’m if I’m P dose so uh III don’t know
00:48:02that’s never happened to us kind of
00:48:04lambda n API gateway allow you to
00:48:07throttle things you can just configure
00:48:08throttling so you can say that you know
00:48:11I want to run up to a thousand
00:48:13concurrent functions of this or up to I
00:48:16think the limit by default is a thousand
00:48:18per function but then you can increase
00:48:21it or with the API gate where you can do
00:48:24throttling based on an API key an
00:48:26authorization key or kind of generally
00:48:29on an API endpoint so you can configure
00:48:31throught link so you can with the
00:48:34monitoring they have and things like
00:48:35that spot if you are kind of being DDoS
00:48:37again my assumption is that Amazon will
00:48:42protect against DDoS much much better
00:48:44than I can code that I don’t know kind
00:48:47of about anybody in the audience whether
00:48:49you feel you can build a better DDoS
00:48:51defense system than Amazon but certainly
00:48:54you can configure it to be to get an
00:48:57early warning and then figure out what
00:49:00to do from there so you don’t have to
00:49:02spend millions and millions and millions
00:49:04if you get the dust at the moment the
00:49:12biggest disadvantage is that lambda
00:49:14functions are limited to about five
00:49:17minutes run
00:49:18so anything that takes longer than five
00:49:20minutes you need to split into multiple
00:49:22executions which means you can’t keep an
00:49:24open socket we were trying to develop
00:49:27something that talks to the Twitter API
00:49:28s and Twitter doesn’t have a push API or
00:49:33you know if you’re on mortal you cannot
00:49:36get the push API you need to connect the
00:49:38socket and get them to kind of stream
00:49:41stuff to you and with lambda that’s not
00:49:43I mean it’s possible but you need to
00:49:45kind of load it every five minutes and
00:49:46then disconnect in the render state so
00:49:48for something like that I would still
00:49:50use ECS another kind of reasonable
00:49:53disadvantage for many people is that
00:49:55there’s this virtually no SLA the lambda
00:49:58or not virtual is actually no isolation
00:50:02and that they don’t offer any date and
00:50:04they don’t of
00:50:04Neela’s yet so our experience is that
00:50:08kind of the u.s. East one the region
00:50:13that is overloaded with everything
00:50:14because that’s the first one that
00:50:15started occasionally gets kind of
00:50:17hiccups where we get a bit of delay but
00:50:21we’ve never really had a full outage
00:50:23since February 2016 when we started kind
00:50:28of moving to this that doesn’t it’s not
00:50:29going to happen and I think you know as
00:50:32Lomb that becomes more and more
00:50:33important I assume they will start
00:50:35providing a slice for it at some point
00:50:38so that’s an interesting limitation B I
00:50:40guess those would be those would be the
00:50:46two key key limitations for people so I
00:50:49think we ran out of time I don’t know if
00:50:51we have kind of I’ll be around you know
00:50:54you can pick me up in in in in the
00:50:56corridor and then we’ll talk about this
00:50:57thing more I need to get other people to
00:50:59set up thank you very much
00:51:00[Applause]
”