The Mesmer CLI can pull data about your Mesmer tests into TestRail.
If you name a Mesmer test containing one or more TestRail testcase IDs (like
C1234), the CLI can match up your Mesmer tests with your tests in TestRail.
By running the apply command, you can create a new TestRail run containing results for each testcase which mirror those in Mesmer.
To populate a new TestRail run with results from Mesmer, you can use
testrail apply. This will connect to your TestRail instance and create a new test run containing results from a given Mesmer build.
# Configure environment (or pass as arguments, see `--help` for details)$ export MESMER_TESTRAIL_URL="https://your-testrail-instance.com/"$ export MESMER_TESTRAIL_USERNAME="firstname.lastname@example.org"$ export MESMER_TESTRAIL_API_KEY="xxx" # generated in settings, or just the account password.$ export MESMER_TESTRAIL_PROJECT=2 # your TestRail project ID, found in its URL # Create a TestRail Run, named after the current date, in the# TestRail project with ID 1.$ mesmer integration testrail apply \ --build $build \ --map 'tag(SKIPPED)::Custom Status' ✓ Fetched TestRail configuration TestRail results we're about to add:Mesmer: Test ID, Name, Tags, Status TR: Case, Status60de7f491959380013988312 [C5] Always Fails Failed ➝ C5 Failed60de7ee619593800139882fc [C3,C4] Email sign in Passed ➝ C3 Passed60de7ee619593800139882fc [C3,C4] Email sign in Passed ➝ C4 Passed60de802f1959380013988372 [C6] Never Run SKIPPED Review ➝ C6 Custom Status Rules we used to generate these results:(highest precedence first)Mesmer TestRailtag(SKIPPED) ➝ Custom StatusPassed ➝ PassedFailed ➝ FailedError ➝ BlockedNeeds Review ➝ Retest Create a TestRail Run with these results? yes ✓ Created TestRail Run: http://your-testrail-instance.com//index.php?/runs/view/8 ✓ Mesmer results successfully applied
This assumes your TestRail instance is configured in single-suite mode, which is the default for new installations. For legacy, multi-suite configurations, you can pass the
--tr-suite parameter to specify the ID of a test suite.
To run the command non-interactively (like from a CI build), you can pass the
-y flag to skip the confirmation prompt and always create the test run.
- First, we fetch some metadata from your TestRail instance, including all the custom statuses you may have configured.
- Then, we generate a rule list, which maps from Mesmer test metadata (like status and test tags) to your TestRail statuses. You can override anything in the rule list with the
- We fetch the Mesmer test results for the build you specified.
- For each result, we find any relevant TestRail cases by looking for
C###patterns in test names.
- For any Mesmer test with a relevant TestRail case, we calculate the right TestRail status by picking the first applicable entry from the rule list.
- If running interactively, we'll stop here and prompt for confirmation---so you can confirm that the automation is doing the right thing.
- Then, we'll create a new TestRail run containing all the results.
By default, the CLI will use a few mappings between Mesmer test status and TestRail default statuses:
Passed ➝ PassedFailed ➝ FailedError ➝ BlockedNeeds Review ➝ Retest
You can add your own rules to this list, or override the defaults, with the
Mappings look like this:
Passed::Passed # Map passed tests in Mesmer to TestRail 'Pass' statusesError::Blocked # Map Mesmer errors to TestRail 'Blocked' statuses
Mappings have two halves, separated by the
:: mark. The left specifies a filter for a Mesmer test, and can take a few different forms:
PassedFailedErrorNeeds ReviewNewtag(some tag) # specify any test tagged with "some tag"
On the right, you can specify a TestRail status, like so:
PassedFailedBlockedRetestSome Custom Status # custom status named "Some Custom Status"label(Some Custom Status) # custom status named "Some Custom Status"system(status_system_id) # custom status with 'status_system_id' as the system identifier
Put together, you can write mappings like this:
# For tests tagged in Mesmer as 'Skipped', post results to TestRail# with the custom status "Mesmer Skip"tag(SKIPPED)::Mesmer Skip # For tests which pass in Mesmer, post results using a TestRail status# with the system identifier "some_custom_status"Passed::system(some_custom_status)
You can add mappings with the
--map parameter, which can be repeated:
$ mesmer integration testrail apply \ --build $build \ --map 'tag(SKIPPED)::Custom Status' --map 'Passed::system(some_custom_status)'
Mappings given first take priority over those passed later.
In certain TestRail configurations, we might need to look for a test case named
C1234, rather than a test case with ID
1234. In this situation, use the
$ mesmer integration testrail apply \ --build=$build \ --resolve-cases-by-name
apply command will fetch all TestRail cases in your current project, and match Mesmer tests to then based on their names, rather than their IDs.
Note that this option will most likely make the integration slower, as this forces the CLI to make several paginated requests for every test case in your TestRail project. If you use TestRail Cloud, this may also cause you to hit the API rate limit, if you have several thousand test cases.