XRAY: explaining the (complex) relations between Test Plans, Tests, and Test Executions

Have you heard about the synch between Test Plans (TP) and Test Executions (TE)?

Don’t feel bad if you didn’t – there isn’t any. Not full synch in any case.

This can be very confusing and important to understand in regard to data integrity.

What to do to avoid data integrity issues?

Verify that all the tests in the test section of the TP are also embedded in TE in the same TP.
- If all TE tests run (Pass, Failed, etc., not in TODO) you will see unlinked tests in the TP as TODO.
  In the TP choose Create Test Execution=>with Status=>TODO and run them in the new TE.
Verify that Tests appear only in one TE associated with the same TP.
If you remove TE from a TP because it is not relevant, you need to manually remove the tests in the TP.
Also important: Test executions are in context to TP: only results from the embedded TE will be seen in the TP.
- Test X Failed, ran again later, and passed.
  If the Fail is n a TE linked to a TP and the same test passed in a different TE, the TP will show: Fail.
- If you run the same test in different TE in the same TP – the last result is seen.

Tests result in the TP example:

Tests in a TP:

TE in a TP:

State	Action	Result
Add tests to a TP and add all the tests to TE.	Remove tests from the TE (the test doesn’t exist in other TEs in the TP).	In the TP: TE remains w/o the Test. All the tests remain in the TP. Tests that had resulted in the removed TE are set to TODO.
Add the test to a TP and then add all the tests to TE.	Remove tests from the TP	The tests remain in the TE.
		The result of running the tests in the TE will NOT be seen in the TP.
		In the XRAY TP report, only the results from the tests in the TP are presented – not from the TE! TP results = only TP tests TP Report - only tests from the TP:
Add a new TE to a TP with tests	*and one of the tests failed	In the TP you see the tests from the TE. In the TP you see the failed test as failed (as in the reports).

An example that can be confusing

You see below the reports of TE and TP.

All the TEs are related to the TP.

The report below (#1) shows TP, in this case, it includes Tests w/o TE in status TODO.

But in the upper report (#2) that shows only TE you have results that include the tests run in the TE (w/o the tests in TODO).

The # of tests therefore is also different – 765 in the TPs and 738 in the TE.