Optimizing Skye Editor using JetBrains dotTrace

This post is a transcript of an internal post I did on Innoveo Solutions internal blog. Thanks to Innoveo to let me share this here!

Skye Editor is our metal model editor which is written in C# 4, WPF uses Model-View-ViewModel design pattern and MVVM Light.

The post shows the usage of JetBrains dotTace to optimize Skye Editor and the importance of profiling your code, here it is.

For the release 2.20 of our Skye Editor product we have done already some optimization like “FindProductValue of ModelProduct to use a dictionary”

Starting of the release 2.21 the goal was to go one step further with “Optimize Loading/Deleting of definition and update to MVVMLight 4 RTM”

The results are quite awesome!

Here I am comparing the last version of Skye Editor which we shipped, 2.18 to the version currently in development for the next release 2.21.

The performance measurement scenario is as following:

• Starting the application
• Loading a big definition, BigDefinition.zip, 2743 Kb zip, 19928 Kb Xml
• Deleting a brick which as lots of sub bricks and attributes, value ranges, values...

I used the profiler dotTrace from JetBrains to measure the performance improvement.

Here is a first result for the method ActualizeFromNewArchive, which is used when we load, import or activate a definition. This method is responsible of building all the View Models used in the editor which we use to display the tree of root, brick, the attributes, value range, values but also the backendinfo.. and finally the texts. So on big definition there is a lot to create especially for the texts.

2.18

2.21

So we went from 9083ms to 944ms which is around a 9.6 factor improvement as we can see on the following picture!

That's quite impressive. But where does it come from? Let dig deeper in the execution tree.

2.18

2.21

So the first improvement is due an improvement done on MVVM Light 4 RTM a library we are using from the beginning which lets us decouple our View Models / Views using some messaging mechanisms among other features. I helped it's author Laurent Bugnion to test and to improve the toolkit, he even mention us on MVVM Light 4 RTM.

We went from 2219ms to 35ms but across the whole scenario (all usage of the Register method) we went from 3072ms to 130ms, which we can see here:

The improvement there is that CleanupList is not anymore done at that moment but only when the application is idle. Clever. And what is really cool is that I mentioned that performance issue and Laurent fixed it in the next release. Thanks Laurent!

But this is not all because we have won only 3072ms which doesn't bring us from 9083ms to 944ms.

The other big improvement is the optimization of the way we find value which as radically changed.

2.18

2.21

From 5103ms to 13ms !

2.18

        public ProductValue FindProductValue(string uuid)
        {
            return Values.AsBindingQueryable().FirstOrDefault(pv => pv.UUID == uuid);
        }

2.21

        public ProductValue FindProductValue(string uuid)
        {
            return Values.FindBindingByUuid(uuid);
        }

Look more in details

2.18

        private readonly List<TBinding> _bindingList;

        public IQueryable<TBinding> AsBindingQueryable()
        {
            return _bindingList.AsQueryable();
        }

2.21

        public TBinding FindBindingByUuid(string uuid)
        {
            Tuple<TBinding, TModel> value;
            _modelDictionary.Value.TryGetValue(uuid, out value);
            return value != null ? value.Item1 : default(TBinding);

The huge difference between those two methods is that the 2.18 is using a list and LINQ to find the first value which match the uuid we are searching. When the 2.21 is using a dictionary which index all values by uuid.

Another improvement of the 2.21 was to go from the following version of the method to the previously shown one:

        public TBinding FindBindingByUuid(string uuid)
        {
            return _modelDictionary.Value.ContainsKey(uuid) ? 

                   _modelDictionary.Value[uuid].Item1 : default(TBinding);
        }

This one make two access to the dictionary and the other only one access, which improved also quite a bit.

Another improvement is that we removed the usage of a ThreadSafeObservableCollection which was dispatching to the UI thread all operations. Basically you could operate the collection from a background thread while it was bound to the UI, which normally you cannot do due to thread affinity, except if you dispatch, which for sure as a cost.

So that's it for the improvement when we load/import/activate a definition!

Now about deleting.

2.18

2.21

It would be nice to have this gain but in fact we had to refactor the operation so that one part is executed on the UI thread on the other part on a background thread. So basically what touch to the View Model is executed into the UI thread and what touch the Model on the background thread.

So we have also to count this

2.18

2.21

So we go from 30720ms to 21666ms. Which is again a good improvement

This can be again improved a lot because currently we have to traverse the whole tree and count all the relations to the texts we want to delete which is accounting for 19468ms.
With a cache of relation it will much much faster. But that for next time!

I hope you will enjoy the time saving of all those optimizations in Skye Editor!

Using Thread.Sleep() in Unit Test! A good idea?

In my humble opinion it is definitely not a good idea! Why?

1. It is brittle test because it depends to the CPU load of the machine running the test. Maybe it runs fine on your development machine, and will for sure from time to time fail on your build server because of the load on the server.
2. It is slower then needed. If you increase the sleep time so that you “ensure” that the test should pass in all situation then the test will always as long as this time.

So what can we do about it?

One of the first approach is to using polling like NUnit and it’s Delayed Contraint does. But I am not a big fan of this because you have to remember the details of it, and you can fall in that trap easily:

Use of a DelayedConstraint with a value argument makes no sense, since the value will be extracted at the point of call. It's intended use is with delegates and references. If a delegate is used with polling, it may be called multiple times so only methods without side effects should be used in this way.

My personal preferred approach is:

1. to expose as a protected property the Task running the background operation
2. to create a SUT class in my test class which inherit from the class with the protected property
3. to make the Task a public property of the SUT class
4. to have my test use the Task by calling the Task.Wait() just before the assertion

For example here is one of my test using that approach:

Test Driven Development Kata - String Calculator

The subject of this article is the String Calculator proposed by Roy Osherove. The Kata is done in C# uses Visual Studio 2012, JetBrains Resharper 7 and NCrunch.

But what is a TDD Kata? It is an implementation exercise starting with unit tests and leveraging refactoring that you practice daily for around 15 to 30 minutes.

The goal of this video is to present one of those session I did recently of this practice that I really find interesting.

The video was also published on Tech Head Brothers, my french portal about .NET.

Microsoft Bluetooth Mobile Keyboard 6000 for developers

I love my Microsoft Bluetooth Mobile Keyboard 6000. I have one at home and one at work.

But there is two things which I don’t really like as a developer:

1. No contextual menu key
2. Two keys combination for Home and End

So I searched for a solution and found one at least for the contextual menu key. The solution is named SharpKey.

SharpKeys is a utility that manages a Registry key that allows Windows to remap one key to any other key. Included in the application is a list of common keyboard keys and a Type Key feature to automatically recognize most keyboard keys.

This little tool let me remap the Caps Lock, which no one use right? to the the contextual menu key, which makes my developer life much easier as I don’t need to move my hand from the keyboard to the mouse to get to that menu.

Great improvement!

At the moment I don’t have found a better way for the Home and End key, but I am considering remapping the volume up and volume down.

TeamCity and Gitlab working together with SSH Keys

On February I posted about “Running your TeamCity builds from PowerShell for any Git branch” and now I have to configure a new ssh key so that TeamCity can connect to Gitlab.

I struggled a bit getting continuous error message from TeamCity that the connection failed because it was enable to load identity file. File rights were all ok on the key file and everything looked fine. But still the error message.

Then I remembered that I had the same issue in February while using PuTTYgen to generate my SSH key pair.

I forgot a minor details that I had to generate the key pair like this:

1. Click Generate button and move your mouse
2. Click Save public key
3. Click Save private key (if you need it in this format)
4. Click on the menu Conversions / Export OpenSSH key and save that key

Then upload the exported OpenSSH key on the TeamCity server. The one in the putty format isn’t working, and that was my problem!

Finally copy the public key generated to your clipboard

and use it in Gitlab

Now when you test the Git connection from TeamCity you should see the following

Automate your Jabbr chat using Github Hubot all running on a Windows machine

I previously talked about “Running Github Hubot on a Windows machine”, “Using Github hubot and Appharbor service hook to get deployment status in Campfire/HipChat rooms” or even “Starting TeamCity builds from HipChat using Github Hubot”.

All those solution rely on an external chat service either Altassian HipChat or 37signals Campfire.

If you want to have your own all integrated experience with all your tools running on premise I propose to look how you might do that using Jabbr and Hubot on a windows machine.

Jabbr is a chat application built with ASP.NET and SignalR. It is open source, available on Github, and provided by David Fowler. Easy to install too as you will see.

To install it the only thing you will need to do is to clone it from Github. You might use Github for Windows then you are one click away from having the clone on your local machine, just visit Jabbr Github page and click Clone in Windows button.

Next step is to create a database named Jabbr, for example in Sql Express.

Finally run the build.cmd from the Jabbr cloned folder which will produce a site folder in the target folder of the Jabbr cloned folder.

Then leverage WebMatrix by right clicking ‘Open as a Web Site using WebMatrix’ and finally click Run.

You should end up with your own local installation of Jabbr, something like the following

Then create a user using /nick [user] [password]

Now let’s create a Jabbr room which will be joined by our Hubot later on.

You just need to use the Jabbr command: /create [room]

Let’s use Development as the name of our room.

Now we are ready to install the second part of our solution Hubot.

Hubot comes with the notion of adapters which makes it very flexible toward what system it can connect to. In our case we would like to have Hubot connecting to Jabbr so we need a Jabbr adapter. Thanks to Scott Moak for creating one which is available here.

But before we need to to install nodejs and Hubot, just follow my previous post “Running Github Hubot on a Windows machine” which is still working with nodejs v0.8.2.

Update: After a short discussion with Tom Bell (@thetombell) on Twitter it is clear that I made too complicated. Thanks Tom for pointing out the easiest way!

So here it is!

mkdir hubot-local

cd hubot-local

npm install hubot

cd node_modules\hubot

npm install https://github.com/smoak/hubot-jabbr/tarball/master

So now we need to integrate Scott Jabbr Hubot adapter. To achieve that from the same prompt your installed Hubot, you will need to install one dependency of the Jabbr adapter which is njabbr so go on and type

npm install njabbr

Now you would need to clone Hubot Jabbr adapter and get the file src/jabbr.coffee and copy it in the adapter /hubot/src/adapters folder of hubot.

Finally just adapt the /hubot/src/robot.coffee to add the new Jabbr adapter like this

HUBOT_DEFAULT_ADAPTERS = [ "campfire", "shell", "jabbr" ]

 

And the last step is to configure some environment variable used by the Jabbr adapter so that hubot connects to the Jabbr chat application and room.

In this example I used the excellent WebStorm from JetBrains but you could do it directly from your environment.

Update: if you don’t have WebStrom you might use PowerShell and

$env:HUBOT_JABBR_PASSWORD = ”HubotPwd”

$env:HUBOT_JABBR_ROOMS = ”Development”

$env:HUBOT_JABBR_HOST = ”http://localhost:17221”

Update: Settings the environment variables like shown previously uses a process-level environment variable (that is, an environment variable that is only visible to, and lasts only as long as, your current PowerShell session). To create more permanent environment variables (i.e., user-level or machine-level) you need to use the .NET Framework and the SetEnvironmentVariable method

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_PASSWORD", "HubotPwd", "User")

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_ROOMS", "Development", "User")

[Environment]::SetEnvironmentVariable("HUBOT_JABBR_HOST", "http://localhost:17221", "User")

Now the only differences from what we have seen in the previous article "“Running Github Hubot on a Windows machine” is that we need to run hubot with the following command line specifying that we want to use the Jabbr adapter:

node .\node_modules\coffee-script\bin\coffee .\bin\hubot –a jabbr

And again the magic starts

As you can see we have Hubot joining our Jabbr Development room and replying to our command. For sure all our Hubot scripts are working, so it lets me check and start builds on TeamCity, but also see AppHarbor deployment status. It is very really easily extendable with new hubot scripts written in CoffeeScript. So it let you customize and automatize most your daily processes. A big win!

Using Github hubot and Appharbor service hook to get deployment status in Campfire/HipChat rooms

Appharbor provides a service hook which let’s you get informed when a build is finished sending both succeeded and failed builds.

We wanted to use this to get informed in our Campfire / HipChat room so that our distributed team can be informed about builds status without having to go to AppHarbor web site.

The idea was to extend our GitHub Hubot hosted on Heroku.

From AppHarbor support here is the information we need about “Developing a service hook”

We will send a POST request content-type "application/json" and the following body:

{
  "application": {
    "name": "Foo"
  }, 
  "build": {
    "commit": {
      "id": "77d991fe61187d205f329ddf9387d118a09fadcd", 
      "message": "Implement foo"
    }, 
    "status": "succeeded"
  }
}

We need to build a Hubot script; which are CoffeeScript, to have an HTTP endpoint listening to this Post payload. Then it needs to read the payload and format it to be able to send it in readable message to the Campfire / HipChat room.

With the release 2.1.3 of Hubot there is a new easy way to have an HTTP Listener:

HTTP Listener

Hubot has a HTTP listener which listens on the port specified by the PORT environment variable.

You can specify routes to listen on in your scripts by using the router property on robot.

module.exports = (robot) ->
  robot.router.get "/hubot/version", (req, res) ->
    res.end robot.version

There are functions for GET, POST, PUT and DELETE, which all take a route and callback function that accepts a request and a response.

We can use this easily in our Hubot script which is called appharbor-listener.coffee.

module.exports = (robot) ->
  robot.router.post "/hubot/appharbor", (req, res) ->
    robot.logger.info "Message received for appharbor"

Now that we are able to listen to POST payload on the url …/hubot/appharbor we need to send a message to the Campfire / HipChat room, which is a bit different from the other scripts. The http listener scripts doesn’t get msg which is normally used to send the response from our bot to the room. Here we have to do it differently and use robot.send which I found on the post ‘Hubot HTTP Daemon Support’

user = robot.userForId 'broadcast'
user.room = 'Your Room Id'
user.type = 'groupchat'

message = "AppHarbor build '#{buildStatus}' for application: '#{builtApplicationName}'"

robot.logger.info "User: '#{user.room}','#{user.type}'"
robot.logger.info "Message: '#{message}'"

robot.send user, "#{message}"

Currently this is working only with the Campfire adapter, the HipChat one is crashing as described here.

Here is the whole script

And finally here is the result of posting a sample payload using fiddler

Running Github Hubot on a Windows machine

I finally managed to get Github Hubot running on my Windows 7 64 bits machine. Thanks to  Sean Copenhaver & Thomas Kahlow which pointed me to the correct direction on the ‘Run hubot on windows’ discussion.

So here is how I did it.

First of all download and install nodejs 0.6.15. This will also install npm 1.1.16 which is the node package manager.

Then start Powershell and create a new folder e.g. HubotWorking

mkdir HubotWorking

cd .\HubotWorking

npm install hubot

You should see the following output

Go on with the following

cd .\node_modules\hubot

node .\node_modules\coffee-script\bin\coffee .\bin\hubot

And the magic!

You might want also to run it through JetBrains WebStorm 4.0, so here is the configuration you will need

And here is the result in the WebStorm nodejs runner

Sending your Git branch changes as an email attachment

The other day I wanted to send per email some code to a friend which doesn’t use Git. He is using Svn and I use Git Svn in front of this Svn repository. Why I do that? Don’t get me started…

So he couldn’t pull from my repo and we were kind of stuck. Really?!? For sure not, here was the goal I set as I am sure this will happen some other time: having the computer work for me. What a strange idea you would say! Yeah, the computer working for you. At the end aren’t we here to make the cool things and let the computer do the boring things?

Ok so I came up with a quick hack. I wanted to have a way to send all the new, or modified, files of my current Git branch per email as a zip attachment. Guess what it was quite easy even for a PowerShell beginner like me.

First of all I needed first to be able to determine on which Git branch I was curently. I googled and find the following

function Get-GitBranch {
    $symbolicref = git symbolic-ref HEAD
    $branch = $symbolicref.substring($symbolicref.LastIndexOf("/") +1)
    return $branch
}

Then I wanted to be able to Zip all the files, but to achieve I had to determine which were the files to Zip. this is done using the following

git diff --name-only HEAD..master

And making a Zip out of the list of files is done like this

function Zip-GitBranch([string]$zipFilename) {

    $branch = Get-GitBranch
   
    if (!$zipFilename) {
        $zipFilename = [string]::Format(".\{0}.zip", $branch)
    }

    $files = git diff --name-only HEAD..master

    foreach($file in $files) {
         & 'C:\Program Files\7-Zip\7z.exe' a $zipFilename $file
    }
   
    return $zipFilename
}

Finally I wanted to be able to send the zip as an attachment of an email using Outlook

function MailZip-GitBanch($Recipient) {
   
    if (!$Recipient) {
        Write-Host "You need to pass the email of the recipient as parameter"
        return
    }
   
    $branch = Get-GitBranch
    $zipFilename = [string]::Format(".\{0}.zip", $branch)
    $attachement = [IO.Path]::GetFullPath( $zipFilename )

    Zip-GitBranch($attachement)
       
    $ol = New-Object -comObject Outlook.Application
    $Mail = $ol.CreateItem(0)
    $Mail.Recipients.Add($Recipient)
    $Mail.Subject = "Changes for the branch: " + $branch
    $Mail.Body = "Check out the email attachement to see the changes made to the branch: " + $branch
    $Mail.Attachments.Add($attachement)
    $Mail.Send()
}

Now I can type the following command to send my changes to my friend

MailZip-GitBranch myemail@email.com

You can find the whole script on the following gist.