Xamarin

Cross-Platform UI Testing with Specflow, Xamarin, and CodedUI: Part 1

Matt Wilhelm

• 5 min read

Just about every developer is familiar with the benefits and costs of Test-Driven Development.  Regardless of your personal stance on the topic, it is hard to argue against the evidence that automated testing often pays huge dividends in improved quality, reliability, and efficiency.  In fact, on most well-managed projects these days unit testing is a first class citizen.

In modern development it is critical to understand our use cases and requirements (even as they are changing) to reduce rework, and to give us the greatest probability of actually meeting the needs of our users.  As project stakeholders, we spend considerable effort trying to understand the problem domain; and as developers we try to demonstrate our understanding by writing clear and concise tests that ensure our systems behave as intended from the get-go.   To that end, we as developers are in a constant struggle with other stakeholders to communicate unambiguously.

When developing applications, we often want an easy way to confirm our work meets the business requirements without having to invoke more meetings, or consume the cycles of other people unnecessarily.  Essentially, we want to smoke test our app or feature before moving on to user testing.  Fortunately, there are many great frameworks to do this, especially for the web. However, on a recent Xamarin project I wanted to set up UI testing for Android, iOS and Windows UWP.  In my search for automated testing Nirvana, I came across a number of solutions – none of which seemed to work for what I was trying to accomplish.

Microsoft maintains a project called Coded UI which brings UI testing to Windows platforms.  It has been used for some time in the MS community and even has a built-in test generation feature that can take much of the legwork out of writing UI tests.  Of course, as soon as a requirement changes, the tests have to be regenerated, which can lead to a management nightmare. Xamarin also maintains Xamarin.UITest, which is convenient for testing Android and iOS apps, but it doesn't work with Windows platforms.  So we are left with two similar tools which are seemingly incompatible…that is - until you add an amazing open source library to the mix called SpecFlow.  SpecFlow allows you to write behavioral tests using the Gherkin specification language, and with some work it is supported by both CodedUI and Xamarin.UITest.  Gherkin is platform agnostic and lets us define the behavior of our app in plain language – so now all we have to do is turn it into code!  Sounds easy right?!

There are many great blog posts out there to help you understand how these tools work; and you should read them – because in this article I will be stitching together these methods to combine CodedUI and Xamarin.UITest with SpecFlow across platforms and in a way that is friendly to refactoring and Continuous Integration.

You should start with Rob Gibbens’ excellent article about BDD Tests with Xamarin.UITest and SpecFlow .  It is what really inspired this effort and is the backbone of this process.  A big shout out to Rob for saving me hours of figuring out how to do all of this on my own. Also, the SpecFlow documentation is quite helpful.  Check it out here.

So now to the TL;DR:

In this example, I will be using Visual Studio 2015 with Xamarin installed, and I will be connecting to a mac on my network for the iOS portions.  You will also need to install the SpecFlow for Visual Studio 2015 extension.  Unfortunately the development story around sharing code between Windows Platforms and Apple platforms is still somewhat complex.  Hopefully Microsoft’s recent acquisition of Xamarin can minimize that.

I will also be using the MvvmCross-Samples TipCalc sample application to demonstrate the process.  I chose this solution because it includes iOS, Android and UWP projects; and because it shows one way of sharing code across platforms while still using the native user interface.  You can download the source from the MvvmCross Github, or you can follow along with my completed solution which is forked from the same repo.

So now, we are going to set up our project.  The high-level process is as follows:

  1. Load the example solution (or your solution)
  2. Set up a regular Shared Class Library (.Net Framework 4.5, not a PCL)
  3. Set up a SpecFlow custom generator for CodedUI
  4. Set up CodedUI and Xamarin test class libraries
  5. Build the framework infrastructure
  6. Create build scripts that can be executed from your CI environment

Throughout this series, I'll be refering to a Github repo.  For the impatient among us, head on over to Github to see the completed solution.

Things to Consider

There are a few "Gotcha's” along the way that I'll list here in no particular order.

  1. CodedUI is highly dependent on MSTest while Xamarin.UITest uses NUnit. They mix like water and oil.
  2. In order to execute these tests, you'll need access to a Mac to execute the Xamarin.iOS portion and a Windows instance (a VM will work) to execute the CodedUI tests. I actually use 2 machines, but a mac with parallels will work.
  3. Because of the dependency on two different platforms and test frameworks, we'll need to make sure all shared code is properly abstracted - so we can make a nice vinaigrette out of our water/oil base.
  4. For this example, we will only be developing tests for Android, iOS, and Windows UWP, but this process can work for testing Windows Phone, 8.1, or WPF apps as well. The TipCalc sample app has projects for these other platforms.  Go ahead and try it out.

Strategy

The crux of this strategy relies on leveraging Xamarin's excellent work in their Xamarin.UITest library that already takes care of abstracting the platform for us.  In Xamarin.UITest, there is an IApp interface that provides a common language for working with platforms in a fluent way.  Since Xamarin.UITest doesn't work with windows (yet?), we need to implement this interface on Windows and call the respective CodedUI methods.

SpecFlow uses template generation under the hood to translate the Gherkin feature files into .NET code.  Unfortunately because we are required (at time of writing) to use different testing frameworks, we will have to maintain our SpecFlow feature files twice and specify different code generation tools for each.  I’ve found this to be the easiest solution but if you know of another way, let us know in the comments.

Setting up the Shared Class Library project

The Shared project will host all common code that can be used in either the CodedUI project or the Xamarin.UITest project.  Create a new .NET 4.5 Class Library (not a portable class library) project and pull in the SpecFlow and Xamarin.UITest nuget packages.  We're grabbing Xamarin.UITest in the Shared library to gain access to the aforementioned IApp interface.  We are also going to:

  1. Create an abstract base class for our features
  2. Create an abstract base class for our steps
  3. Implement Assert
  4. Add an IScreen interface that can be inherited for each screen we will be navigating to (more on this later)
  5. Add static constant classes to define some magic strings.
  6. Set up a SpecFlow custom generator for CodedUI

Since we are using two different testing frameworks, both of which are based on attributes – we are going to create an abstract base class to share our test initialization and shutdown logic.  We’ll then create an implementation in each platform project that can be properly decorated for the appropriate test framework.  Now let’s create abstract base classes we can use for our SpecFlow features & steps:

link to source

link to source

Now, because we are using two different testing frameworks, we are going to implement our own Assert so we have more control over our testing, and to remove any dependency on Microsoft, NUnit, or Xamarin namespaces.  We also add a class to hold some magic strings.  More on this later.

Set Up SpecFlow Custom Generator for CodedUI

SpecFlow uses code generation to take care of much of the boiler plate in creating feature classes.  Unfortunately it doesn't support CodedUI out of the box, so we need to tell the MSTest generator to add the proper annotations.  By following this wiki post, we’ll create the generator plugin and link it with our windows project – which is created in the next section.  I typically generate the DLL once and then copy it into the packages directory, since this project won’t change.

That's it for Part 1.  Today we discussed the high level strategy and put the basic infrastructure in place we will need.  The key components for handling features and steps are now defined and I hope you see how this project will come together.     Please join us  for Part 2 where we will set up CodedUI and integrate it into the Xamarin.UITest style of device activation.

Follow me on twitter @AddressXception