Continuous Deployment and Continuous Integration are topics that you can evolve and improve forever. The best reading on it IMO is [1] by Jez Humble, and it will be the guide for this post.
"The time from deciding that you need to make a change to having it in production is known as the cycle time, and it is a vital metric for any project."
What we want here is do reduce the cycle time of the project in a reliable, secure and low-risk way, an automated process that is trusted enough to not be rolledback.
It means you are going to need a good deployment pipeline with lots of software testing before making the shine feature in production.
Some other stuff counts here like metrics visibility, easy triggers to start the process, etc.
"Automation is the key. It allows all of the common tasks involved in the creation and deployment of software to be performed by developers, testers, and operations personnel, at the push of a button.
When deployments aren’t fully automated, errors will occur every time they are performed. The only question is whether or not the errors are significant. Even with excellent deployment tests, bugs can be hard to track down. When the deployment process is not automated, it is not repeatable or reliable, leading to time wasted on debugging deployment errors."
Again speed here counts because you cannot make money without the feature or bugfix in production.
Every change (or commit) should trigger the pipeline, the artifact resulted from the CI pipeline should be tested before going to the release.
The principal benefit of the approach that we describe in the preceding section is that it creates a release process that is repeatable, reliable, and predictable, which in turn generates large reductions in cycle time, and hence gets features and bugfixes to users fast. The cost savings alone are worth not just the cover price of this book, but also the investment in time that the establishment and maintenance of such a release system entails.
The best advise of the book: If It Hurts, Do It More Frequently, and Bring the Pain Forward.
It is the way you have a baseline for your servers, how you have the certainty that package X is installed or removed from your system, that the service Y will be UP. It is how you mantain a initial state for you OS in an automated way. We have a lot of CM systems out there: Ansible, Puppet, Chef…
With the tools borned the concept of IaC (infrastructure as a code), you now can model your infrastructure assets in a dependendable fashion, in any environment (bare-metal, private/public cloud), and make sure all boxes remains the same.
To make sure you have this implemented Jez makes a good question: Can you be sure that each deployed environment in production, in stagin and in test is set up in precisely same way? Could you regress to an earlier know good state of your app?
The goal of CI is to keep everyone in sync with each other, which we achieve by making sure that newly checked-in code propery integrates with existing code. [2]
Here you can use tools like TravisCI or Jenkins to merge your fix branches on the master, run all unit tests and create the artifact from it. The trick here is process feedbacks, it is the first part of the pipeline and if you have it very concrete it should be the base to the rest.
When the book were wrote Docker, (2013) didn't existed yet, even the 12-factor methodology (2012), so things in this field are VERY new, as the concept of microservices as well.
People are still discovering and adapting how these processes can work with these new technology. A good example is Deis: a lightweight application platform that deploys and scales 12-factor apps as Docker and containers across a cluster of CoreOS machines.
So to resume: the 12-factor methodology is a composition of the best practices from the CI/CD process for modern application and Deis uses great technologies like CoreOS and Docker to make it practical, on other posts we will analyze how it is implemented.
As stated on Chapter 6 of Building Microservices, deploying a monolithic app is a fairly straighforward process. Microservices, with their interdependence, are a different kettle of fish altogether. If you don't approach deployment right, it's one of those areas where the complexity can make your life a misery.
Make CI/CD systems well integrated with your infrastructure is a complex task, things get worse when these systems must be modularized and easy to adapt to your deploy pipeline. You have a lot of rowing up applications with different requirements, code and ways to run. This layer on top of your infrastructure is now known as PaaS, Heroku is a great example of how you can abstract the complexity of running a new app in 2 or 3 commands. It should be awesome to have the same capability on your hands and control of the stack, this way you can modify or replace pieces in a transparent way. For the company they probably will have to worry about the total cost of ownership and good people to manage it.
Some examples of opensource software in the market to achieve it: Jenkins Pipeline and the guys from CloudBee [1] made a good job with it, you can run slaves as Kubernetes pods for example, and create stages in a very dynamic way for your pipeline.
DEIS workflow and its microserviced architecture [2]: a Heroku SDK for Kubernetes, they have very detailed documentation worth to read, I am going to explain the general parts on this article.
The guys from DEIS have other project pretty much interesting called HELM, it is much more simple than Deis Workflow, think of it as a package manager for Kubernetes.
We are using the "new" kubeadm tool to setup the cluster on a Debian machines for development, take a look on the documentation but you can run this script to have it up:
#!/bin/bash
# Remove old kubeadm stuff
kubeadm reset
systemctl restart kubelet
kubeadm init
if [ -f /etc/kubernetes/kubelet.conf ]
then
export KUBECONFIG=/etc/kubernetes/kubelet.conf
fi
# Create pod network
kubectl apply -f https://git.io/weave-kube
# Enable pods start
kubectl taint nodes --all dedicated-So far so good, you have a bunch of docker containers check it with ps or try to run a new 'kubectl run nginx –image=nginx'
We can start Deis with the following helmc script:
$ helmc generate workflow-v2.8.0
$ helmc install workflow-v2.8.0
NAME READY STATUS RESTARTS AGE
deis-builder-415246326-97ioi 1/1 Running 13 15h
deis-controller-1590631305-rqev9 1/1 Running 13 15h
deis-database-510315365-3oro8 1/1 Running 7 15h
deis-logger-9212198-n38h8 1/1 Running 17 15h
deis-logger-fluentd-mny1o 1/1 Running 4 15h
deis-logger-redis-663064164-jm8fx 1/1 Running 4 15h
deis-minio-3160338312-oxhp9 1/1 Running 4 15h
deis-monitor-grafana-432364990-fieg8 1/1 Running 4 15h
deis-monitor-influxdb-2729526471-d7v66 1/1 Running 4 15h
deis-monitor-telegraf-d3gva 1/1 Running 14 15h
deis-nsqd-3264449345-inu6p 1/1 Running 4 15h
deis-registry-2182132043-5go81 1/1 Running 14 15h
deis-registry-proxy-s6aph 1/1 Running 4 15h
deis-router-2457652422-tznjt 1/1 Running 6 15h
deis-workflow-manager-2210821749-4lqkp 1/1 Running 4 15hTake a look on [3], you can see the flow diagram of how DEIS workflow behaves, and provide to the developer a simple interface for deployments.
The principal idea here is a release. One of the problems faced with raw kubernetes deployments is when you have to update/change your mounted secret on an environemnt variable, it doesn't reflect direct on your pod, so you have to restart or redeploy it manually. With DEIS a release can be a code change or a configuration change, if one of these change a new release is deployed.
Basically DEIS workflow have a gitserver with some hooks ("git-receive-pack", "git-upload-pack"), after you push your git repo to the system, the system detects your build type (dockerbuild or slugbuilds) and create a new pod that builds the artifact, DEIS controller in the end will start a deployment object on your Kubernetes cluster.
The operator frontend is a command line called "deis". On deis/workflow-cli repo, exists a folder called cmd, on auth.go we find the Register func, these files are more a wrapper and the initial glue for the logic business, deis/controller-skd-go has the sdk. IF you follow the stack via auth.go:64 you hit controller-sdk.go auth.Register it calls "v2/auth/register" on your controller.
First of all we need to find the router IP service via "kubectl –namespace=deis describe svc deis-router", and use it to call the register command: "deis register http://deis.10.6.40.159.nip.io".
If you start your domain with the app name like: app.10.6.40.19.nip.io, workflow router will handle it and route to your application. DEIS will label router.deis.io/routable: "true" on each service you want to route packets to.
Going back to the register namespace on the controller, we have a Django (Rest Framework) app with a PostgreSQL database, that stores all configurations.
It is possible to replace parts of your cluster for a development one in a few "make deploy" command. Or you can customize it on the Helm Classic installation.
Going through the flow we find the create command, again take a look on AppCreate and controller-sdk-go/apps, func New, here it makes a POST on Django v2/apps it basically create a new api_apps row on PostgreSQL db. With the app created you can start releases and set other configurations.
After register a new user and login the system, you have to put your publickey on DEIS to be able to push code.
deis keys:add ~/.key.pubWhen you create a new app (deis create yourapp), a remote git repository will be created and point to your builder host (port 2222). All you need to do is push your local repo to the remote:
git push origin deisDEIS Builder will handle your request and based on the server hooks start the builder pod, suppose we are using a Dockerfile commited on the project, lets take a look on the dockerbuild file:
...
# The .tar.gz file artifact is fetched from the Builder Storage
with tarfile.open("apptar", "r:gz") as tar:
tar.extractall("/app/")
# Use docker api to build and push docker to registry
client = docker.Client(version='auto')
registry = get_registry_name()
imageName, imageTag = os.getenv('IMG_NAME').split(":", 1)
repo = registry + "/" + os.getenv('IMG_NAME')
stream = client.build(tag=repo, stream=True, decode=True, rm=True, pull=True, path='/app')
log_output(stream, True)
print("Pushing to registry")
stream = client.push(registry+'/'+imageName, tag=imageTag, stream=True)
...PS: In production I really recommend to use ECR (external docker repository) and an external artifacts storage, take a look on ~/.helmc/workflow/charts/workflow-v2.8.0/tpl/generate_params.toml, otherwise you can full your disk very fast:
storage = "s3"
repository_location = "ecr"
[ecr]
region = "us-east-1"
hostname = "...ecr.us-east-1.amazonaws.com"
[s3]
registry_bucket = "deisregistry" # Docker Registry
database_bucket = "deisdb" # PGSql backups
builder_bucket = "deisbuild" # ArtifactsPS2: You can have a similar JOB on jenkins pipeline, something like:
node {
stage 'login'
sh '$(aws ecr get-login)'
stage 'git'
dir('project') {
git credentialsId: 'xxx', url: 'git@github.com:comp/proj'
}
stage 'build_push'
docker.withRegistry('http://..us-east-1.amazonaws.com', 'proj') {
def proj = docker.build("proj:${env.BUILD_NUMBER}", ".")
proj.push()
}
}I have the artifact and now? Step back and ask again: I have the artifact, how? The answer is the func build on pkg/gitreceive/build.go, it started the build pod watched it run outputed the logs and launched it with hook.CreateBuild, here we came back again to the controller pod c.Request("POST", "v2/hooks/build", b), can you see what da heck if microservices?
urls.py
url(r'^hooks/build/?$', views.BuildHookViewSet.as_view({'post': 'create'})),
To resume the stack- > BuildHookViewSet -> starts a new Build object and with a post_save signal starts a new release with models/build.py:65
self.app.deploy(new_release)
models/app.py:446 as a FK of the build. lets deploy a new revision of the app.
tasks = [
functools.partial(
self._scheduler.deploy,
namespace=self.id,
name=self._get_job_id(scale_type),
image=image,
entrypoint=self._get_entrypoint(scale_type),
command=self._get_command(scale_type),
**kwargs
) for scale_type, kwargs in deploys.items()
]
async_run(tasks) # Just a bunch of asyncio threadsThe _scheduler object here looks interesting, the AuditModel (parent of App) do importlib.import_module(settings.SCHEDULER_MODULE) on scheduler property, the scheduler module starts the KubeHTTPClient and the deploy creates/update a deployment Kubernetes via REST API with some custom data.
A dashboard [5] can be useful for log analysis for example, DEIS comes integrated with fluentd for log and grafana + influxdb for metrics and monitoring.
Helm 2.0 [6] is Kubernetes package manager build by DEIS, you can find some ready Charts, kudos to the Jenkins one:
helm install stable/jenkinsYou can start your own helm with create and manage the deployments with upgrade, releases and etc. Another good project on top of kubernetes to keep the eye on.
DEIS Workflow provides a very cool PaaS environment for the end lazy developer, normally peopl don't give a shit about the stack behind their code, and just want to run it. So DEIS is the option with some custom configuration and an extended dashboard you can provide a good self-service model of deployment.
HELM is a Kubernetes package manager, a bunch of templates with custom configuration where you can replicate the "deployment" in a few commands, you don't have any fancy stack to start things, but is awesome to automate and reply your software in a ease way.
Follow the Sig-APPS community for the latest news.