Skip to main content

mesmer integration testrail

The Mesmer CLI can pull data about your Mesmer tests into TestRail.

If you name a Mesmer test containing one or more TestRail testcase IDs (like C1234), the CLI can match up your Mesmer tests with your tests in TestRail.

By running the apply command, you can create a new TestRail run containing results for each testcase which mirror those in Mesmer.

mesmer integration testrail apply#

To populate a new TestRail run with results from Mesmer, you can use testrail apply. This will connect to your TestRail instance and create a new test run containing results from a given Mesmer build.

# Configure environment (or pass as arguments, see `--help` for details)$ export MESMER_TESTRAIL_URL="https://your-testrail-instance.com/"$ export MESMER_TESTRAIL_USERNAME="testrail-service-account@yourdomain.com"$ export MESMER_TESTRAIL_API_KEY="xxx" # generated in settings, or just the account password.$ export MESMER_TESTRAIL_PROJECT=2     # your TestRail project ID, found in its URL
# Create a TestRail Run, named after the current date, in the# TestRail project with ID 1.$ mesmer integration testrail apply \  --build $build \  --map 'tag(SKIPPED)::Custom Status'
 โœ“  Fetched TestRail configuration
TestRail results we're about to add:Mesmer: Test ID,         Name,                   Tags,      Status     TR: Case,  Status60de7f491959380013988312 [C5] Always Fails                  Failed  โž  C5         Failed60de7ee619593800139882fc [C3,C4] Email sign in              Passed  โž  C3         Passed60de7ee619593800139882fc [C3,C4] Email sign in              Passed  โž  C4         Passed60de802f1959380013988372 [C6] Never Run          SKIPPED    Review  โž  C6         Custom Status
Rules we used to generate these results:(highest precedence first)Mesmer                        TestRailtag(SKIPPED)               โž  Custom StatusPassed                     โž  PassedFailed                     โž  FailedError                      โž  BlockedNeeds Review               โž  Retest
Create a TestRail Run with these results? yes โœ“  Created TestRail Run: http://your-testrail-instance.com//index.php?/runs/view/8 โœ“  Mesmer results successfully applied

This assumes your TestRail instance is configured in single-suite mode, which is the default for new installations. For legacy, multi-suite configurations, you can pass the --tr-suite parameter to specify the ID of a test suite.

To run the command non-interactively (like from a CI build), you can pass the -y flag to skip the confirmation prompt and always create the test run.

How it Works#

  1. First, we fetch some metadata from your TestRail instance, including all the custom statuses you may have configured.
  2. Then, we generate a rule list, which maps from Mesmer test metadata (like status and test tags) to your TestRail statuses. You can override anything in the rule list with the --map parameter.
  3. We fetch the Mesmer test results for the build you specified.
  4. For each result, we find any relevant TestRail cases by looking for C### patterns in test names.
  5. For any Mesmer test with a relevant TestRail case, we calculate the right TestRail status by picking the first applicable entry from the rule list.
  6. If running interactively, we'll stop here and prompt for confirmation---so you can confirm that the automation is doing the right thing.
  7. Then, we'll create a new TestRail run containing all the results.

Adding Mappings#

By default, the CLI will use a few mappings between Mesmer test status and TestRail default statuses:

Passed        โž  PassedFailed        โž  FailedError         โž  BlockedNeeds Review  โž  Retest

You can add your own rules to this list, or override the defaults, with the --map parameter.

Mappings look like this:

Passed::Passed      # Map passed tests in Mesmer to TestRail 'Pass' statusesError::Blocked      # Map Mesmer errors to TestRail 'Blocked' statuses

Mappings have two halves, separated by the :: mark. The left specifies a filter for a Mesmer test, and can take a few different forms:

PassedFailedErrorNeeds ReviewNewtag(some tag)  # specify any test tagged with "some tag"

On the right, you can specify a TestRail status, like so:

PassedFailedBlockedRetestSome Custom Status          # custom status named "Some Custom Status"label(Some Custom Status)   # custom status named "Some Custom Status"system(status_system_id)    # custom status with 'status_system_id' as the system identifier

Put together, you can write mappings like this:

# For tests tagged in Mesmer as 'Skipped', post results to TestRail# with the custom status "Mesmer Skip"tag(SKIPPED)::Mesmer Skip
# For tests which pass in Mesmer, post results using a TestRail status# with the system identifier "some_custom_status"Passed::system(some_custom_status)

You can add mappings with the --map parameter, which can be repeated:

$ mesmer integration testrail apply \  --build $build \  --map 'tag(SKIPPED)::Custom Status'  --map 'Passed::system(some_custom_status)'

Mappings given first take priority over those passed later.

Resolving by Name#

In certain TestRail configurations, we might need to look for a test case named C1234, rather than a test case with ID 1234. In this situation, use the --resolve-cases-by-name flag:

$ mesmer integration testrail apply \  --build=$build \  --resolve-cases-by-name

Then, the apply command will fetch all TestRail cases in your current project, and match Mesmer tests to then based on their names, rather than their IDs.

Note that this option will most likely make the integration slower, as this forces the CLI to make several paginated requests for every test case in your TestRail project. If you use TestRail Cloud, this may also cause you to hit the API rate limit, if you have several thousand test cases.