TeamCity and Gitlab working together with SSH Keys

On February I posted about “Running your TeamCity builds from PowerShell for any Git branch” and now I have to configure a new ssh key so that TeamCity can connect to Gitlab.

I struggled a bit getting continuous error message from TeamCity that the connection failed because it was enable to load identity file. File rights were all ok on the key file and everything looked fine. But still the error message.

Then I remembered that I had the same issue in February while using PuTTYgen to generate my SSH key pair.

I forgot a minor details that I had to generate the key pair like this:

1. Click Generate button and move your mouse
2. Click Save public key
3. Click Save private key (if you need it in this format)
4. Click on the menu Conversions / Export OpenSSH key and save that key

Then upload the exported OpenSSH key on the TeamCity server. The one in the putty format isn’t working, and that was my problem!

Finally copy the public key generated to your clipboard

and use it in Gitlab

Now when you test the Git connection from TeamCity you should see the following

Continue Reading »

An article by Laurent Kempé No Comments

Automate your Jabbr chat using Github Hubot all running on a Windows machine

I previously talked about “Running Github Hubot on a Windows machine”, “Using Github hubot and Appharbor service hook to get deployment status in Campfire/HipChat rooms” or even “Starting TeamCity builds from HipChat using Github Hubot”.

All those solution rely on an external chat service either Altassian HipChat or 37signals Campfire.

If you want to have your own all integrated experience with all your tools running on premise I propose to look how you might do that using Jabbr and Hubot on a windows machine.

Jabbr is a chat application built with ASP.NET and SignalR. It is open source, available on Github, and provided by David Fowler. Easy to install too as you will see.

To install it the only thing you will need to do is to clone it from Github. You might use Github for Windows then you are one click away from having the clone on your local machine, just visit Jabbr Github page and click Clone in Windows button.

Next step is to create a database named Jabbr, for example in Sql Express.

Finally run the build.cmd from the Jabbr cloned folder which will produce a site folder in the target folder of the Jabbr cloned folder.

Then leverage WebMatrix by right clicking ‘Open as a Web Site using WebMatrix’ and finally click Run.

You should end up with your own local installation of Jabbr, something like the following

Then create a user using /nick [user] [password]

Now let’s create a Jabbr room which will be joined by our Hubot later on.

You just need to use the Jabbr command: /create [room]

Let’s use Development as the name of our room.

Now we are ready to install the second part of our solution Hubot.

Hubot comes with the notion of adapters which makes it very flexible toward what system it can connect to. In our case we would like to have Hubot connecting to Jabbr so we need a Jabbr adapter. Thanks to Scott Moak for creating one which is available here.

But before we need to to install nodejs and Hubot, just follow my previous post “Running Github Hubot on a Windows machine” which is still working with nodejs v0.8.2.

Update: After a short discussion with Tom Bell (@thetombell) on Twitter it is clear that I made too complicated. Thanks Tom for pointing out the easiest way!

So here it is!

mkdir hubot-local

cd hubot-local

npm install hubot

cd node_modules\hubot

npm install https://github.com/smoak/hubot-jabbr/tarball/master

So now we need to integrate Scott Jabbr Hubot adapter. To achieve that from the same prompt your installed Hubot, you will need to install one dependency of the Jabbr adapter which is njabbr so go on and type

npm install njabbr

Now you would need to clone Hubot Jabbr adapter and get the file src/jabbr.coffee and copy it in the adapter /hubot/src/adapters folder of hubot.

Finally just adapt the /hubot/src/robot.coffee to add the new Jabbr adapter like this

HUBOT_DEFAULT_ADAPTERS = [ "campfire", "shell", "jabbr" ]

 

And the last step is to configure some environment variable used by the Jabbr adapter so that hubot connects to the Jabbr chat application and room.

In this example I used the excellent WebStorm from JetBrains but you could do it directly from your environment.

Update: if you don’t have WebStrom you might use PowerShell and

$env:HUBOT_JABBR_PASSWORD = ”HubotPwd”

$env:HUBOT_JABBR_ROOMS = ”Development”

$env:HUBOT_JABBR_HOST = ”http://localhost:17221”

Update: Settings the environment variables like shown previously uses a process-level environment variable (that is, an environment variable that is only visible to, and lasts only as long as, your current PowerShell session). To create more permanent environment variables (i.e., user-level or machine-level) you need to use the .NET Framework and the SetEnvironmentVariable method

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_PASSWORD", "HubotPwd", "User")

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_ROOMS", "Development", "User")

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_HOST", "http://localhost:17221", "User")

Now the only differences from what we have seen in the previous article "“Running Github Hubot on a Windows machine” is that we need to run hubot with the following command line specifying that we want to use the Jabbr adapter:

node .\node_modules\coffee-script\bin\coffee .\bin\hubot –a jabbr

And again the magic starts

As you can see we have Hubot joining our Jabbr Development room and replying to our command. For sure all our Hubot scripts are working, so it lets me check and start builds on TeamCity, but also see AppHarbor deployment status. It is very really easily extendable with new hubot scripts written in CoffeeScript. So it let you customize and automatize most your daily processes. A big win!

Continue Reading »

An article by Laurent Kempé 7 Comments

Sending your Git branch changes as an email attachment

The other day I wanted to send per email some code to a friend which doesn’t use Git. He is using Svn and I use Git Svn in front of this Svn repository. Why I do that? Don’t get me started…

So he couldn’t pull from my repo and we were kind of stuck. Really?!? For sure not, here was the goal I set as I am sure this will happen some other time: having the computer work for me. What a strange idea you would say! Yeah, the computer working for you. At the end aren’t we here to make the cool things and let the computer do the boring things?

Ok so I came up with a quick hack. I wanted to have a way to send all the new, or modified, files of my current Git branch per email as a zip attachment. Guess what it was quite easy even for a PowerShell beginner like me.

First of all I needed first to be able to determine on which Git branch I was curently. I googled and find the following

function Get-GitBranch {
    $symbolicref = git symbolic-ref HEAD
    $branch = $symbolicref.substring($symbolicref.LastIndexOf("/") +1)
    return $branch
}

Then I wanted to be able to Zip all the files, but to achieve I had to determine which were the files to Zip. this is done using the following

git diff --name-only HEAD..master

And making a Zip out of the list of files is done like this

function Zip-GitBranch([string]$zipFilename) {

    $branch = Get-GitBranch
   
    if (!$zipFilename) {
        $zipFilename = [string]::Format(".\{0}.zip", $branch)
    }

    $files = git diff --name-only HEAD..master

    foreach($file in $files) {
         & 'C:\Program Files\7-Zip\7z.exe' a $zipFilename $file
    }
   
    return $zipFilename
}

Finally I wanted to be able to send the zip as an attachment of an email using Outlook

function MailZip-GitBanch($Recipient) {
   
    if (!$Recipient) {
        Write-Host "You need to pass the email of the recipient as parameter"
        return
    }
   
    $branch = Get-GitBranch
    $zipFilename = [string]::Format(".\{0}.zip", $branch)
    $attachement = [IO.Path]::GetFullPath( $zipFilename )

    Zip-GitBranch($attachement)
       
    $ol = New-Object -comObject Outlook.Application
    $Mail = $ol.CreateItem(0)
    $Mail.Recipients.Add($Recipient)
    $Mail.Subject = "Changes for the branch: " + $branch
    $Mail.Body = "Check out the email attachement to see the changes made to the branch: " + $branch
    $Mail.Attachments.Add($attachement)
    $Mail.Send()
}

Now I can type the following command to send my changes to my friend

MailZip-GitBranch myemail@email.com

You can find the whole script on the following gist.

Continue Reading »

An article by Laurent Kempé 2 Comments

Starting TeamCity builds from HipChat using Github Hubot

After writing about “Running your TeamCity builds from PowerShell for any Git branch” I’d like to talk about another integration which is using Github Hubot so that the build could be started directly form a chat room.

So using the same idea, which is to extend our development environment, we implemented a way to start TeamCity builds directly from our HipChat room. You could do the same from Campfire for sure.

The main idea of this is to have our daily tools right at our disposal where we spend a good part of our days, chat rooms.

First we had to install Github Hubot and we have chosen to use Heroku which offer free hosting for one of their web dyno. They even offer what they call Hubot Factory which is a way to allow people to easily and quickly deploy new instances of Hubot to Heroku. In fact we went with a manual installation described on this page “Hubot for HipChat on Heroku”.

The idea is still the same we need to run TeamCity builds by “Accessing the Server by HTTP”.

This time we want to have Hubot calling the TeamCity server so we need to add a new CoffeeScript to the scripts of Hubot which will launch the HTTP request needed.

Our script is heavily inspired from the TeamCity.coffee script, and here is a first version:

# Starts a build on TeamCity.
#
# You need to set the following variables:
# HUBOT_TEAMCITY_USERNAME = 
# HUBOT_TEAMCITY_PASSWORD = 
# HUBOT_TEAMCITY_HOSTNAME = 
#
# start build buildId -- Starts TeamCiyt build using buildId
module.exports = (robot) ->
  robot.respond /start build (.*)/i, (msg) ->
    username = process.env.HUBOT_TEAMCITY_USERNAME
    password = process.env.HUBOT_TEAMCITY_PASSWORD
    hostname = process.env.HUBOT_TEAMCITY_HOSTNAME
    buildName = msg.match[1]

    msg.http("http://#{hostname}/httpAuth/action.html?add2Queue=#{buildName}")
      .headers(Authorization: "Basic #{new Buffer("#{username}:#{password}").toString("base64")}",
               Accept: "application/json")
      .get() (err, res, body) ->
        if err
          msg.send "Team city says: #{err}"
          return

By the way I started to learn CoffeeScript so this is more hacking then something really productive.

And here is the result in our HipChat room asking hubot to start a build with the command:

hubot start build bt21

I will talk in another post about the last part which shows the status of the build in the chat room.

Continue Reading »

An article by Laurent Kempé No Comments

Running your TeamCity builds from PowerShell for any Git branch

I love TeamCity and use it since a while to automate my build/release processes. As human we should never do the work a machine can do, we have certainly better and more interesting things to do.

The habit I saw in the different projects I worked for is to create new TeamCity builds for the branches you work on. It take quite some work to do, even with templates…

So I came with another way of doing it. It is leverage Git, PowerShell and the possibility to run TeamCity builds by “Accessing Server by HTTP”. Basically you just need to make a HTTP request to an Url like this:

http://testuser:testpassword@teamcity.jetbrains.com/httpAuth/action.html?add2Queue=bt10&system.name=<property name1>&system.value=<value1>&system.name=<property name2>&system.value=<value2>&env.name=<environment variable name1>&env.value=<environment variable value1>&env.name=<environment variable name2>&env.value=<environment variable value2>

We will also use the possibility to pass custom parameters (system properties and environment variables) through the HTTP request.

The parameter we want to pass is the Ref name of the Git VCS Root, so we start by setting it using a configuration parameter called BuildRefName, as this:

Then we need to add a configuration parameters for our TeamCity project with the name BuildRefName which we currently set to master, so that if the parameter is not defined it will default to this value:

Now that we have configured this you can already start a build from your browser by calling the URL!

http://testuser:testpassword@teamcity.jetbrains.com/httpAuth/action.html?add2Queue=bt10

Be sure to replace bt10 with your build id.

Nice ,but we want to get a step further and be able to start the build from PowerShell, which is quite easy!

To achieve this goal you add those two functions to your PowerShell profile:

function Get-Web($url, 
    [switch]$self,
    $credential, 
    $toFile,
    [switch]$bytes)
{
    #.Synopsis
    #    Downloads a file from the web
    #.Description
    #    Uses System.Net.Webclient (not the browser) to download data
    #    from the web.
    #.Parameter self
    #    Uses the default credentials when downloading that page (for downloading intranet pages)
    #.Parameter credential
    #    The credentials to use to download the web data
    #.Parameter url
    #    The page to download (e.g. www.msn.com)    
    #.Parameter toFile
    #    The file to save the web data to
    #.Parameter bytes
    #    Download the data as bytes   
    #.Example
    #    # Downloads www.live.com and outputs it as a string
    #    Get-Web http://www.live.com/
    #.Example
    #    # Downloads www.live.com and saves it to a file
    #    Get-Web http://wwww.msn.com/ -toFile www.msn.com.html
    $webclient = New-Object Net.Webclient
    if ($credential) {
        $webClient.Credentials = $credential
    }
    if ($self) {
        $webClient.UseDefaultCredentials = $true
    }
    if ($toFile) {
        if (-not "$toFile".Contains(":")) {
            $toFile = Join-Path $pwd $toFile
        }
        $webClient.DownloadFile($url, $toFile)
    } else {
        if ($bytes) {
            $webClient.DownloadData($url)
        } else {
            $webClient.DownloadString($url)
        }
    }
}

function tcBuild([string]$branch) {

	$url = "http://YourServerUrl/httpAuth/action.html?add2Queue=YourBuildId"

	if ($branch) {
		$url = $url + "&name=BuildRefName&value=" + $branch
    }
	
	$credentials = New-Object System.Net.NetworkCredential("Username", "Password")

	Get-Web $url -credential $credentials
}

Please adapt the tcBuild script by replacing the YourServerUrl, YourBuildId, Username, Password by your personal values.

The first function Get-Web is from this blog post: “Microcode: PowerShell Scripting Tricks: Scripting The Web (Part 1) (Get-Web)”.

The second, tcBuild, is quite easy and is the one you will use to start a build on the TeamCity server.

My workflow now is the following one:

1. I commit on my local Git repository then
2. when I want to run a build I push to the centralized Git origin repository in my branch then
3. I start my build on the Git Branch from PowerShell by typing :

> tcBuild Jobping-10-NewFeature

where Jobping-10-NewFeature is the name of my Git branch.

This is quite cool and give me also the possibility to run kind of personal builds!

Continue Reading »

An article by Laurent Kempé 2 Comments

DVCS and my usage of Git Svn

Sometime ago I moved “away” from subversion has my Version Control System (VCS) because I felt not free of my way of working with it. I found a great was to improve my work experience by using Git Svn in front of our central Subversion repository.

I started my experimentation with Mercurial and Hg Svn because I was already using Mercurial for my personal projects. I hit some much the wall with Hg Svn that I decided quite quickly to go and try Git Svn. I had no experience with Git at that time and I had the feeling that it was more complex to handle than Mercurial (which I still think).

So currently I use Git Svn to work on one of our product at Innoveo Solutions. I also made a presentation to the team during our techno pizza lunch about DVCS, Git, Git Flow and Git Svn.

And here is the list of what I personally gained as a developer:

1. 2 steps commit
   1. Stage/commit, Push
2. Local history / branching
   1. No connection to central repository needed to branch/to look at history
3. Experiment / Refactoring / Spikes
   1. Commit changes on one path, if wrong rollback
4. Smarter Merging
   1. Git’s focus on content rather file location
   2. Better at resolving merge conflicts for you (e.g. renames)
   3. Branching/Merging is daily workflow not anymore an ‘exceptional case’
5. Stash changes
6. Rebase / Rewriting history
   1. Until push you can use interactive rebase

Due to my 4h daily commute (more than 2h in the train), I very much appreciate the offline capabilities of DVCS. It lets me branch, watch the history without having a connection to the central repository. So that a really convenience thing in my day to day job.

But what I even prefer is working in local branches for experiment, refactoring. With that capability I am able again to commit small steps of code change. I particularly appreciate the possibility to experiment some refactoring and to rollback wrong changes when I feel that I went the wrong way with the change. In the past this was not possible and often you would not commit to SVN because that code would be shared with the others or t would break the build. Ok, I know I could do that with Svn working in a feature branch, but we all know the pain it is with the merging back, especially when you refactor and rename files. During refactoring it is important to be able to save/commit small chunk of code change and even more important is to be able to rollback those changes. So in that case DVCS is a perfect match to that problem.

What I currently don’t like in my current way of working with Git in front of Svn is that I have local branches which I don’t sync back to the central server. I don’t like it because if I have an issue with my local machine then those changes might be lost. I will investigate in the coming weeks about possible solutions even if the best one would be a migration of the central repository to Git. But his is another story because it means a wider change in the team.

Continue Reading »

An article by Laurent Kempé 2 Comments