As part of our ever increasing effort to automate more of the builds and tests that we run on our applications and thereby detecting defects as early as possible, integrating our UI testing solution Ranorex with our TeamCity CI server became more and more important.
When using TeamCity to build your application and testsuites, you run into an interesting problem; TeamCity by default runs under the Windows System account and, by extension, does not have access to an interactive desktop. Ranorex, because of it’s very nature, needs a desktop to interact with.
Several solutions float around the internet, but they all boil down to allowing your TeamCity service on the agent running the test to run as a local user, a user that is also logged in to the machine and has a visible desktop.
And it is this final sentence that provided me with the most headaches. Our buildagents in TeamCity are all virtualized and our systems people do not allow RDP access to the agents.
PsExec is a small command line utility provided by Microsoft as part of their Sysinternals suite. It allows you (provided you have the right credentials, the executable is available on the target machine, etc.) to run any executable on a remote machine, and have it interact with the desktop. As Ranorex can build a command-line testrunner application, and use it’s floating license available on the network, we can add a normal machine to our buildprocess with a vanilla Windows install on it.
This command line runner can generate a .rxlog file with the results of the testrun and will exit with an exitcode indicating success (0), failed test (-1) or other errors (any positive number). Sadly, this exitcode is not propagated by PsExec, so we had to find a different solution for letting the testproject fail when the tests fail.
Our new setup
Anonymised, our new setup looks kind of like this:
- The BuildApp project in TeamCity delivers an installable artifact to the BuildAndRunTests project
- The BuildAndRunTests builds the ranorex test from the sources.
- A batch file that does the following is started on the agent:
- Copy the installer to the testrunner machine
- Copy also the testsuite
- And copy a batch file to the testrunner
- Start the copied batch file on the testrunner using PsExec
- Await the run of the test and copy all results back to the buildagent for further parsing
- Parse the log from the command line and make the build in TeamCity fail or succeed
- Copy the .rxlog and .rxlog.data files to the Reports subfolder folder in the Artifacts folder of the project and rename them to *.html and *.html.data
The batch file on the testrunner has it’s own responsibilities
- Use the installer to install the version of our application we want to test
- Redirect standard output to a log file
- Start the Ranorex test runner
- Read the exit code of Ranorex and output this to the logfile as well
This means that we can parse the log from the testrunner machine using e.g. powershell and set the exit code based on the original exit code of the Ranorex testscript. When our UI tests fail, we see the result within minutes of a commit. Moreover, since TeamCity is capable of integrating custom html files as build-reports, we can view the results of our testsuite immediately in our CI server.