Build And Test With Jenkins

Hey,
I didn't write for a long time :( Sorry for that, I dont have any time since I started a new Job. I have a lot of things in the drafts, but I dont find the time to finish it. Because of that, I wanted to write about on of the first things I faced when developing code in a proper agile software company - Jenkins.



In my new job I encountered this thing called Jenkins, which is a opensource automated continuous integration and deployment tool. The key thing that as a developer, I can push my code for example into Git, and Jenkins automatically detects it, compile the code, test it, and if everything works, can deploy it to the test server for the other developers to work with. Sounds pretty cool.

In time, we started to grow in both developer numbers and in code complexity. People started to relay heavily on Jenkins during the development phase, and then started the "Build is broken" shouts in the office. What does it mean? For example if I push to the Git a new feature\bug fix, I push it to one branch called "master" which is the branch that every developer push code to. If Jenkins find a problem with the code (not compiling\tests failed) it does not deploy it to the development server. Now, no one can deploy anything to the development server because someone pushed bad code. The entire development process was stuck, sometimes for days as the issue can be more complex to fix. Also, code review was a nightmare, as there was not enforcement to review, and no one can tell what really pushed or not. Its time for order!

I nagged my boss for some decent source control system. I remember in my past working with Github\Gitlab\Bitbucket with great success. I started by testing all the systems, and had the best impression from GitHub. As regulation in our company back then, we had to use in-premise solutions for code. So I convinced my company to buy GitHub Enterprise which is on premise GitHub appliance that I can deploy in our network.

Now the integration started with GitHub and Jenkins. There are few Jenkins\GitHub plugins. The plugins basically let the Jenkins alter the Commit Status on GitHub based on each stage in Jenkins so the developer can see the status of the of the Pipeline directly in the GitHub. On its own its not really solving the issue we had with broken builds. That is when GitHub Pull Requests come to the rescue. Pull Requests are basically the developer says to the company - "Hey, I have some new code, please merge it into the codebase", in our case "master" branch.  The repository maintainer can then comment on the code,  and make code review, and can decide whether to merge or not.  Using a cool plugin called GitHub pull request builder plugin the maintainer can see the entire pipeline for each pull request. Because each PR is managed by its own, only reviewed, tested code is merged into the master branch, causing almost zero build fails.

This thing changed our life! I cant possibly imagine working without PRs. In some scenarios like UIs we made a Jenkins job which deployed the specific PR UI to a separate place, and linked the commit status to the UI URL. The UI developers can showcase their test UI in a real system, without having to break the master one. The repository maintainer can browse the UI, make some tests and only then approve it and merge it. Super cool!

Later on I integrated a thing called reviewable which is the best review tool I ever used, and its great boost for our GitHub Enterprise review system.

And yet, not all is good. That is when we started to face a new issue. Our build\test servers started to explode when we wanted to make a new release for example as all the developers wanted their code merged, and made tons of pushes to the Git, crumbling our infrastructure to the point the developers need to wait a lot of time for the Jenkins to give them build slot.

So with zero experience with those kind of problems, me and another friend from work started developing a solution in AWS. It was basically a auto scale which launched EC2 instances based on the Jenkins build status. Later I found very similar code to our in the internet :(
https://github.com/widdix/aws-cf-templates/blob/master/jenkins/jenkins2-ha-agents.yaml
Because we use GitHub Enterprise I also launched GithHub instance in our AWS, and configured to replicate our On-Premise (no more stupid regulations) so that the AWS machines can clone faster the code.

Thats when I first faced AWS and how awesome this platform is. Which probably deserves some posts of its own. This solution was working decently. But I hate dealing with VMs. I find it clumsy, and a lot of work. I all in for serverless.  I started reading more and more about AWS and found some interesting things.  First thing that I found that can make a better solution using containers. So I read about container, which sounds great and the perfect solution only to find out the container in AWS are backed only by EC2, which is controlled by autoscale, which is basically the same thing we are doing right now, so it wont really help us the boot time of the machines and other clumsiness of handling VMs. Few days before this research AWS released a container service called Fargate, which basically containers that are manged by AWS and not EC2, but it was released only few days before my research, no plugins for Jenkins, no examples at all, and I found it really difficult to use.

I kept reading and found a service called AWS codebuild, which is AWS saying give us container image, a script, and we run it for you. Perfect. So I make good POC with the Jenkins build for our Linux build and NodeJS testing. This was working great, I dont have to deal with VMs! Its super scale, no matter how much people make push to the Git, no one have to ever wait for a build slot. The disadvantage is that AWS codebuild does not support windows, so for windows we have to stay in the old solution.

We still have a lot of problems with our CI system, for example we have tons of jobs in Jenkins, and its impossible to track dependencies between jobs, and changing things in jobs become impossible as all the other jobs stops working, how artifacts change between jobs and so on. I am still searching for possibly better solution than Jenkins but for now with not much success.