[Music]
hello everyone and welcome back to the
kia ora theater
uh at linux conference australia 2022 uh
next up we've got ian weinend uh ian has
been involved with the open with
openstack in the open devca laboratory
for several years as an employee of red
hat and has admin privileges for these
projects and commit privileges for the
broader ci ecosystem of openstack's
devstack project and the zul prodigating
system
uh today he's here to talk about opendev
a collaboratory for open source software
development at scale and how its
infrastructure components come together
to run the services used by thousands of
developers to develop key parts of the
open source ecosystem
ian is taking questions today so make
sure you put them into the questions tab
and venulis throughout
already
uh hello thank you for having me here
today
um
will you be interested in this talk uh
we'll be talking about sort of
continuous integration speculative
testing
things that fall under the category of
continuous delivery cd devops git ops
uh zuul the cicd system
uh
not the netflix one not the ghostbusters
one
um
merge trains pipelines things like that
this will come together with a bit of
ansible containers
and
we're working in a gitish environment so
if you use any of these sort of gitish
code review systems
you might find something in here for you
too
uh
let me pick off the last bit of the
title of the talk first the open dev
collaboratory what what is that
uh many of you will be
uh more familiar with the openstack
project openstack's been going for
12 years something like that it's i mean
it's basically the way to run a sort of
traditional cloud of virtual machines on
top of your hardware
uh
open source of course
um
what what tended to happen with
openstack though was over all these
years
uh collected a lot of projects that
weren't sort of related to openstack
so every problem in computer science can
be fixed with the level of abstraction
we started out what's called the opening
for a foundation and split these
separate projects out so openstack is
now one of these projects under the
opening for a foundation
so what's open dev
opendev is the uh sort of section of
this that provides the
infrastructure services for all these
projects including now openstack
that's the code hosting
code review system garrett all the ci
systems all the collaboration systems
mailing lists esa pads messaging all
that sort of stuff all of that
infrastructure is provided by opendev
to support these projects
uh the way that we work
uh
opendev provides all these services
to these projects which build these
um
uh products such as openstack and cadder
and zul and airship
these are then consumed by a wide range
of companies many of these companies
then contribute back to us
so
vex host for example and rackspace
provide us with a lot of our hosting
for hosting our services
uh lenaro provides us with a lot of arm
64 resources in various ways uh other
providers provide us with uh
hundreds of ci nodes that we used for
running ci for the projects
and all these companies provide
developer time as well so
developers that help out and help run
the infrastructure
we make the infrastructure better the
projects make themselves better the
companies get better it also works in a
in a nice circle
so that's what i that's what the open
dev collaboratory is and that's
the the services that i'm talking about
today are the services provided by the
open dev collaboratory
open infrastructure what do i mean by
open infrastructure
uh well
our goal is to build on open source
technology from top to bottom that's
that's not everybody's goal and we're
we're sort of privileged to have this
goal that we only use open source
technology in our stack
what we want with our infrastructure is
for anybody to be able to turn up and
contribute to our infrastructure
without any explicit permission or
having to join the admin group or having
to be part of a special group that can
can see the infrastructure
sort of having your infrastructure out
in the open isn't
isn't really that common even in open
source projects you know
it's sort of common to maybe do this at
a project level where maybe
you know you have the you have the
website checked in to get and people can
update the website and get and
you push a change and then you know a
little bit later a con job pulls it in
and updates the web page or something
like that but all of the infrastructure
behind that all the
um you know
the web services and everything behind
that
these aren't often sort of exposed
publicly and
you know everything from the
configuration to the orchestration
is not often not
you know open for contribution in in a
free manner
um
even some open source hosting
providers that i've seen sort of say oh
we'd we'd love your help to to help you
know add admin our systems send us a
resume
um
so we definitely don't want to want to
be at this level we want to have our
infrastructure completely open
and
that's the systems that we've built and
that i'll be talking about today
uh the only way you can really achieve
this is to have your deployment and
ci
really so close that
that they're almost the same thing we
call it testing
test like production
uh
you know we can accept sort of
unprivileged changes coming in
and and test them and have a really high
level of confidence that when we commit
these changes
that
we're really close to what is going to
be pushed into production
um
and yeah it really allows us to run open
infrastructure by taking this model this
test-like production model
um
a few more concepts to to pick up zul
zul is the ci system that um really
enables
all of this
um hopefully some of you will be
familiar with zul especially if you've
ever contributed to openstack
zuul would have run your ci jobs on the
the change that you contributed to
openstack
um
but it you know zul is a
a continuous integration delivery and
deployment system so it does all of
these things for you and it works on
project gating and
gating is a little bit of a hard concept
to
exp
um explain but we we all have it
intuitively in our head
and this is
you may have the greatest testing in the
world and if you uh commit your change
and run your testing
uh sorry if you propose your change and
run your testing that's fine but if
somebody else commits to the tree in
front of you
then your testing is now invalid
and and we all sort of in intuitively
understand this
and the only way to safely merge our
changes
when we have more than one developer is
to really put those changes into a queue
and merge them sequentially and
you need a system to help you do this
you know you can't have humans sort of
doing this it just doesn't scale
zul essentially solves this and this is
what we call project gating
uh zul's been doing this for you know
well over a decade
uh uh other um systems now you know this
is now called merge trains or merge
queues in other systems as well they
sort of catching up
but this idea of gating that it's
impossible for us to
uh merge a broken change because um
zul is merging the changes for us humans
don't merge changes humans say this
change is okay to merge and then what
zul will do is put them in a queue run
the testing
in the most efficient way manner in the
most efficient manner possible
and
push those changes in for us
so
um
let's get into the the details more i've
sort of gone through all these concepts
but
we need to get into some details here
today i i just want to
use an example of our paste system which
is just one of our absolute simplest
systems but it'd be interesting to just
walk through a couple of changes here
and probably familiar with the paste
system you know you just put some text
in there you paste it in there and you
get back at url and you know you can
share your paste is good for you know
sharing logs and things like that
uh
open infrastructure you know this is all
open you can go to this this website
here and you can see our the paste
service that we run
what i i just want to pull out
it builds a container just got a docker
file there it builds a container nothing
amazing builds a container based on
python 3.8 okay fine all right let's
just keep that in our head
i'm going to propose a couple of changes
here just to show you how this whole
system works together and we're going to
work through the whole thing here
this is a this is uh what you would see
on our review site review
dot open there this is a garrett
instance
um we're looking at the logit um
paste system uh
this is the open changes for for that
and i'm gonna propose a couple of
changes here that we're gonna use as an
example
uh don't worry about the dnm in front of
it i actually propose these to the live
system this has all been done
you know in the live system dnm means do
not merge it's just an indication to
reviewers
you know i'm doing something don't look
at this
um
so the first change we're going to
update the
container to python 3.9 this is
something you definitely want to test
right you're updating a container from
3.8 to 3.9
that could potentially impact your
services you want to test that
and
i'm also going to put on top of that
just
another change just just for fun just to
show how all this cross project testing
works instead of saying new paste we're
going to make it say enter new paste
just go back see where it says new paste
we're going to make it say enter new
paste just just for fun
okay so these two changes are stacked on
top of each other
these two changes depend on each other
you can sort of see that in the way that
garrett has put the changes on top of
each other
so i've proposed these changes i've
pushed them into the code review system
what's going to happen next
this is the the zuul uh interface that
i've extracted here
uh you can see in the queue here i've
just sort of
squished it all down here
um
zul is going to pick up that i've pushed
two changes and it's it's going to start
running some testing you can see it's
going to start running the first test
on the base change and then it's going
to
put the the second change together with
the first change and start running the
testing on
the second change i'm going to call that
the api change because it you know
changes some text which could be
considered an api
all right so zul's going to run its
testing we're going to wait a few
minutes we're going to come back to the
change
a little bit later this is the change
uh i'm just calling out here that you
know we have this relation chain again
uh
zul has run the testing it's verified it
says okay all the tests pass everything
looks good great
and so at this point you're sort of
going okay
thanks ian you pushed the change and you
ran some testing um
we've been doing that for a long time
that's not very exciting
and that's
true um
but what we've tested here is remember
just the the logic the the the paste
service container here
but this this is not what we deploy
there's a lot more that goes
around this service that you know has
has not been tested at this point but we
we we need to test all this stuff in
blue as the open dev you know
people providing this service to make
sure that these changes are ready for
production
um
yeah all these things in blue
um
i i really
the the production side of the testing
here
so
you know
all these all these these blue bits are
what you would call you know system
configuration
so
now we come to testing and deploying
the the two changes that we sort of
discussed before so
you know system configuration we
actually have a repository where we
store our system configuration called
system config
this is the the url for it and this is
where we're going to jump in to start
testing these changes that we've made to
our service to make sure they're ready
for production and ready to be pushed
you know into service
so
i'm now going to propose a change to our
system config repo
uh
now here's the magic
everything's all the magic sort of comes
from this one little line
uh we depend on
that previous change the api change
that
updates the container to python 3.9 and
makes the makes a little change
and what we do in this system config
change is
we update our testing of the paste
service
see you can see down there instead of
looking for the text uh new paste
in the in the text we're going to look
for please enter your new paste that was
the change we made
we used uh
down the bottom there we use a
um
it's called test infra it's it's a very
cool uh project that
allows you to test your infrastructure
like write unit tests that test your
your infrastructure
uh check it out if that's the type of
thing you you need to do
so
um
yeah i'm sorry
just didn't cut the window looks a
little cut off there
um
so we're going to push this change
and
now i i kind of need to explain to you
what's going to happen
when
we push this change and zul picks up
this change
so
when we propose a change uh from any any
of these services uh zuul is sitting
there as it's listening for the changes
that come in and it prepares what we
call uh an executor environment
so
the first thing it will do is is take
the code from uh the proposed changes
and it will
merge them into
code trees that you need to test against
so zool knows the dependencies you know
we we set up that depends on before
and um
zoo will merge all the code that you
need to test
into uh
the
for you automatically
uh the next thing it will do is set up
uh the nodes where you uh want to test
so these are uh ephemeral test nodes
that they're just
assigned and allocated to you
and
they're the nodes that you use for
testing uh
we won't talk about how those nodes are
allocated that's it's done by node pool
a whole separate thing that could be a
whole separate talk we'll just take it
for granted that we get a couple of
nodes to test on as many as we want
now uh the next
at this point right we have a bunch of
nodes waiting for
something to happen on and
we have our executor environment with a
bunch of code that's ready to be tested
so
this is a job for ansible okay
we set up a job infantry inventory that
points to all the test nodes
and then
our ci
jobs are just ansible playbooks they
just run against the test nodes that
have been assigned
this means that there's
your testing is written in ansible your
deployment is written in ansible
you
you know everything is the same it if
you want your testing to be a shell
script well just make answers we'll call
a giant shell script that that's fine
but you can also use all the great
features that ansible has
you know just as easily in your
job playbooks as well
so uh we're at this point and it's it's
sort of over to the testing at this
point so the first thing that the
test jobs do is copy the code over to
the test nodes uh start running the test
nodes uh the the nodes run their tests
they do whatever they need to
do um
then they return their
logs their artifacts whatever you know
has happened on the nodes that they want
to return for storage and
uh reporting back is returned back
zul takes that back
and zul reports the change back that's
that's the sort of high level overview
of what zul does in response to a change
now
i'm gonna jump ahead and actually
describe how we deploy
merge changes from zul uh this will
hopefully make this should make sense
when i get to the next light but just
stick with me because i know it seems
like i'm jumping ahead
so we now have emerged this is how we
deploy our production changes
we have a merged change okay so we trust
this change this change has been
reviewed uh it's been committed it's
ready for deployment
so zuul picks this up
same thing
zul starts up its executor environment
zul grabs the change now this is now
a merged change
it puts it into into the executor
environment
now
the only difference is that then before
instead of an ephemeral test node that
is is going to disappear
we connect to a bastion host
that has permission to log into our
production hosts
so
we set up zul is allowed to to log into
this bastion host
the the code the merged changes uh are
then pushed onto the bastion host and
then the uh playbooks actually run
the configuration through the bastion
host but it end up targeting the
production host
and so that is how we do uh deployment
viazul
uh to our production services zool
actually does our
you know production deployment for us
zool even deploys itself
you know sort of circular
reference there
so
this is our
production deployment
so when we want to test changes
we need to
we need to test like production what
what are we going to do
well it's all exactly the same except
instead of using the actual bastion host
that is allowed to
speak to the actual production host
we just turn those into ephemeral test
nodes
we we use um
the the nodes that are allocated by node
pool
but we really do the same thing so a
proposed change comes in from anybody
who wants to submit it in garrett
and
we actually
do the exactly the same process except
it deploys onto two nodes that
will disappear at the end of the test
and we don't care about
so
this is our simulator production
environment
and
this is really the key to how we do this
um
test like we're in production
and this is why i say that our testing
is really really close to our deployment
um
you know
we tried it we really is that close that
we have a really high level of
confidence that if we've passed testing
that when we push this into production
it's all going to be okay so i want to
dive into a little bit of of how this
actually works give you some actual
details
this is actually
uh
the job the system config run paste this
is the actual zool job that
is going to be testing the logic
container changes and tests the pace
system um
the job of interest
for our situation here
so
this is dual config but you don't even
need to really understand it you can
just see what's going on you can see
that we require the system configuration
that that makes sense and you can see
that we have this a node set defined
here we need the two nodes this is the
two nodes i've talked about we need a
bastion host node and we need the
production host node
and so that's what we have defined here
uh you know some jobs might have three
four nodes you know they might set up a
little test zookeeper system by
themselves and connect up to that you
know all sorts of things but this is
just the absolute simplest case here of
uh the bastion host and just one simple
production host
so
i have i i've sort of
been through this but
it's still probably not making terribly
much sense as to how this actually
um
you know works and and
um pulls in the changes from that that
logic
container image so
there's there's another little thing
here that sort of just sits here it says
requires logic container image
which is um
interesting we need to sort of pull into
that and see how does that work
so
let's let's jump ship let's go back to
logit the paste service this is where
the jobs are defined for the logic page
service
uh it's rather annoying that for some
reason
the uh
some of the pictures aren't showing up
here
we'll just have to go on
so
the
this is uh
a job called um
the
logit build open dev image this is what
actually builds the
paste service container image for us
you can see
this actually provides
the logic container image so before we
had the requires
and here we have the provides
what's really nice about zuul is that
everything
zuul has a sort of hierarchical
job
concept so this open dev build docker
image is actually a
very generic job um
that we sort of instantiate here to to
just build this particular image
um you can see that this has a
dependency on a thing called the build
set registry
um
if you notice when we uh had the job up
before
that the opened the the build of the
container image actually depended on
this build set registry so it was a
build set
the build set is
is this this this group
this whole set of testing that is
happening against this one particular
change
so what the build set registry is is
actually a registry so a registry is a
place that you can
push container images and pull container
images from
it's actually a place that you can push
a container image for a whole set of
testing this this doesn't seem to make
much sense for this particular case but
he's here's something i've pulled out
from uh where we use node pool
so
here's the the registry job
we actually build a node pool image here
in this
and then we use that image
one two three you know
12 or 13 times
afterwards to build all these these
different platforms so
the uh building the image relies on the
registry once the image is built it
pushes it into that local registry and
then all these jobs
uh depend on both of those and they
actually pull
from that local registry so when we push
a change
we sort of build
the image once
we push it into
the local registry
and then we can pull it as many times as
we need with as many tests
and
we don't have to build it multiple times
but
you sort of still might be saying okay
uh but you seem to be running
across two separate projects here
you've got the the logit side where i i
sort of understand you're building
you're building a container image with
these changes that that's fine but how
does this get across and and still get
tested
on the the configuration side
this seems to be something missing here
well uh
just uh two more concepts i swear
um
we have a concept of an intermediate
registry
so
this is actually a shared registry
that
we can push our changes into that sits
that is not ephemeral that sits outside
of our changes
and allows us to pull
the containers into
uh
other projects and to do this uh
cross-project testing
uh that's great that you can push a
change into
uh
a sort of registry for caching but you
actually have to be able to find that
change again
and so we use another part of zul which
is the
artifact handling which is where zul
keeps uh meditator that jobs return
and um
we use that to look up the
container in the intermediate registry
so
let me actually explain how this works
in detail so we we have our proposed
garrett change
this was the two changes we made
uh we built we ran this job that that
built the the container image the build
open dev image that we've been talking
about
that gets pushed to the build set
registry that's what we've been talking
about that can be used by
this jobs that are being tested right
alongside
that change um
but we also push that change into this
intermediate registry
so this is the long the long lived
registry uh it'll sit in here it just
doesn't like a mark sweep clean up it
might sit in here for a couple of days
until it's not used and then you know
the intermediate registry
will get rid of it an intermediate
registry part of
part of zul a service that that
can be run alongside windsurf
but then we also return an artifact from
the job with a tag that that tells us
where in intermediate registry this
container is that we built
so
let's let's sort of have a look at this
here's the the
build of the
testing image that we want to test and
deploy
this is the the zul interface for the
job result
which you can find
via the same seoul.opendev.org
link that i showed before
uh you click on that you'll see
artifacts
and you'll see we have here a
sort of container image artifact that
in in this case this is really not a
human consumable artifact so it makes
more sense if we we just pull it
um on the command line
you'll see that we have this artifact
type and you'll see that it gives us a
reference here so that we know where to
pull it
from intermediate registry
so
this this is how we end up doing the the
cross project testing and pulling in the
changes from the logic container pulling
them in testing them in our system
configuration and making sure that we we
know everything is is ready to go and
ready for production
so
this requires logic container image this
thing that says here what what this is
actually saying when we unpack it in
words it says
zuul i
can you provide me in in the jobs that
i'm about to run a variable that's
called zul artifacts
and
i i want that to have a list of
artifacts
for any job that i depend on so remember
before we said that
this job depends on the changes to the
the logit container image so for any job
that i depend on
get me the artifacts
any job that i depend on that provides
logic container image
give me their artifacts in this uh
variable
so
you don't really have to look
too much but you can it's all there if
you want to start digging into it so
when we run the system config
run paste job so this is the job that is
going to test the actual deployment of
all the the changes in the logic
container
you can actually see here
where it goes and it pulls
uh from that url that that we were
saying before and it's actually grabbed
that
um
from the artifact so
uh
it does that in zul jobs i just wanted
to sort of
pull up the code and show you that
there's
you know there's no magic here you can
sort of
you can sort of see that
all it does is goes through
this is the job that pulls
uh the container from the the separate
project
and it just does a loop through the zuul
artifacts and looks for
container image
container images in the artifacts and it
pulls them into
pulls them locally so we have
effectively pulled in
the container with the changes from the
other project and that's the container
that we are now going to deploy and use
in our system config test which
is going to ensure that
you know we have end to end tested
the changes
in the uh
logit
that we proposed in the
the logic project over there
um
[Music]
so
so we will we will run our our test
deployment here
uh
i just wanted to to pull out and show
you
that we
as part of the results of the job will
drag back
basically all the logs you could need to
debug what's going on um
you know all the the logs from starting
the container and starting the the
services setting up the database all
that sort of thing
and you also have all the
ansible logs that uh deployed these
changes for you so
um
this is just an example of the the
ansible steps that are deploying into
our um
our simulated production environment
and it
will go through all these steps if
there's any problems it it'll
flag it it'll fail you'll be able to go
back you'll be able to debug it there's
no problems all good you know that
you know that the the container was
pulled in
it started we set up everything we set
up
uh everything from
ip tables to apache to docker to
the start scripts the stop scripts
everything and so it's all there it's
all been tested
um
it's uh yeah that's just the ansible
bits and pieces
um
another kind of cool thing that we do uh
because this is a sort of complete
end-to-end test as part of the the paste
test we actually start up a headless
browser and point that at
the service as well
and so you'll see one of the artifacts
is
called screenshots and we'll actually go
and take a screenshot of the service
that we set up in our
production environment uh our ephemeral
production environment
and you know this is the actual
screenshot here you can see that instead
of saying
saying new pace before now it says
please enter your new pace that was the
change we made so we we have sort of 100
confidence that
the changes we made
uh are deployable have worked
uh this is running now the python 3.9
container uh we know that our api change
has has been made has applied
um
you know this is
this is everything we want from our ci
system we have
great confidence that everything is
going to work when we push this into
production
yeah just calling out that we
did the main thing
so
you know
deployment
is really boring that there's
we we run a separate job for doing
deployment so instead of um
you know system config run paste is the
job that runs on the unprivileged
changes that that come in
um once a change has been reviewed and
committed then a different job runs the
infraprod service paste but it
essentially does you know exactly the
same thing
except it targets the production machine
instead of the
ephemeral nodes
so this is actually the the top level
playbook that deploys the paste service
uh you know nothing very exciting there
you can go to our system config recapo
and have a look at it if you like
so
you know it's just two two branches of
the same thing um
when we're using
when uh unprivileged change comes in the
ephemeral bastion host runs this service
uh against the um
ephemeral paste node and that's what
happens
uh
in the production side we run exactly
the same playbook
but just against the production service
so
there's nothing different and there's
very little to go wrong
um
of course you know i'm i'm glossing over
things
of course production is different to
what we can possibly set up in our
um sort of ephemeral testing situation
a couple of things that come to mind
is let's encrypt
or basically you know ssl
um
ssl setup of services and ci you know we
just have self-signed certificates while
in production of course we have
um
authenticated certificates that we use
let's encrypt to get those to
certificates um
so that that sort of changes things
um
the other thing that we have to worry
about in production is that um
so the build set was the the set of
jobs that run against a particular
change that comes in
uh
we do actually have room for
um
parallel jobs to be running it
in wild in in in the wild so in
production we have um periodic jobs that
are running
for example
uh the the let's encrypt example
previously um you know let's encrypt
um
certificates expire after a couple of
months
we we have a periodic job that um goes
through and makes sure that all our
certificates are uh up to date so you
have to account for the fact that
somebody might have pushed the change
while
uh you know one of these periodic jobs
is running in the background to to keep
thing to keep things going
uh zuul actually
um provides ways for you to to deal with
that at the zoo level uh it gives you
semaphores that you can use so jobs
won't start running on top of each other
um
but you know this is something you need
to deal with um
in production
uh
logs is a big one so obviously in the uh
ephemeral
testing situation
uh where you're setting yourself up with
you know just just all test data and on
ephemeral nodes uh we grab all the logs
and everything's just um
put back in and you see it in uci jobs
we don't publish uh the logs for the um
production deployments
um
i mean we would like to but of course
the practicality is that you know
you can leak secrets fairly easily from
from that side of things
um
so
you know it if we focused on auditing
maybe it would be something maybe would
even be good because you know anything
you did leak you have to make sure that
you can rotate
in an instant
uh
you know
it would be good um but it's not
something we have at the moment
um and also not
not everything is sort of completely
fault tolerant in that
we can just sort of
commit a change and push it and have it
deploy
uh straight away
with zero downtime
some of our services like
gta
which you know runs our code browsing
and stuff like that
we do run in a fault tolerant manner
than that other services we don't um so
they have a little bit more manual
interaction
we consider that a bargain and we're
always
working to
basically
take ourselves out of the loop as much
as we can but
there definitely are still points that
sort of
need a little bit more interaction than
we'd like
um
and but it's all solvable i mean i think
that the general principle here of of
testing like production and and pushing
like this
is is something really um quite valuable
um
i just wanted to show an actual
practical example of
how sort of someone who came along and
contributed to our infrastructure uh
that didn't have
permission as such what wasn't sort of
part of the existing
group of people that generally uh work
on such things
this was just standing up at a service
the refstax service
uh you can check out the change yourself
uh you know we went through a lot of
back and forth you know there was a lot
of learning for everyone
um
47
you know different reviews
but you know because of the way that we
we set up this system you can push your
changes you can see the changes
we you we could
essentially you know
have our sort of simulated production
environment running these changes
and the developer got a lot of feedback
about you know how to how to build up
the the deployment steps
and then by the time we got to the end
after all the reviews
all that was really left to do was
commit and push the change and off it
went and you know popped itself into
production and
we sort of knew it would work because
putting it into production was really so
close to what we've been testing
that
it just sort of it just works
uh yeah this is a good example of how
our open infrastructure works to allow
people to get in and
help us
build and maintain the services that we
provide
so if you're sort of interested
in finding out how this all actually
works it you go and check out zul zulu
ci there's talks there
uh there's specific zuul talks that go
into a lot more detail on zuul and how
it all works
vex host is a big supporter of opendev
they offer a managed zool service
uh and the other thing to check out is
the um software factory project
from red hat
uh that that's not so much um
focused on this sort of cd side of zul
like this but if you think that this is
sort of interesting technology then
software software factory can really
help you get a garrett and a zool and
all these other services down there sort
of
deployed in a really all-in-one manner
it's a great way to sort of
set up a
software factory i guess
[Music]
so
that's it i'm about out of time thank
you
um yeah i just wanted to sort of give
you a bit of an insight into how we
do this open infrastructure that runs on
top of an open open source stack
how this ability to do this cross repro
testing and this speculative testing
and to test like we deploy it really um
allows us to have this open
infrastructure
and
hopefully you can see that having this
open string open infrastructure it's
just such a false multiplier that allows
other contributors to come in and help
you
manage your own infrastructure and for
the people that are sort of
more in the weeds of it
the sort of core reviewers who are going
to be looking at the the changes coming
in it just takes so much load off them
knowing that the ci
is so good that
it's going to be picking up
any of the problems
before they even have to look at it
to chat with us uh opendev on oftc irc
or a matrix uh there's a zuul channel on
matrix
uh you'll find plenty of people who'd be
happy to talk to you
all about it and
that's it
cool thank you so much uh ian that was a
really interesting talk uh we do have
one question which will pop into the
text chat afterwards if you want to join
that um
but
yeah thank you so much next up is our
lunch break uh so come back at 1 30 p.m
uh in this theater we've got michael
cohen with velociraptor digging deeper
in linux for dfir
investigation
thank you so much for your talking have
a great rest of your day
thank you