How to use the Testing Notebook to test a package without shadowing?

From my understanding of what's going on, Mathematica does the following when you hit "Run":

1. Parse the entire test notebook to look for input test cells and get the corresponding cell ids.
2. Run the tests and collect the outcomes (using the cell ids from step 1).
3. Generate test stats (total tests run, successes, failures) and provide links to jump to the next failed test (using the cell ids from Step 1).

Step 1 is the cause for the problem you see, because symbols are generated in the appropriate context at parse time, not when they're evaluated. In this case, the current context at parse time is Global` not test `. To see what I mean, try running the following test notebook:

It will fail at the 4th test. As a workaround, you can do "Evaluate Notebook" instead of "Run" — trying this on the above notebook (after quitting) will fail the 3rd test, but pass the 4th as we want. A major drawback is that you don't get the test statistics and cannot quickly jump to failure points, which can be useful for large test suites.

As a matter of good practice and working within the limitations of the testing framework, I would suggest using the full context for your package symbols in the test cells so that there is no ambiguity.

What R.M. answer means is that Get as a part of the test will not help. The reason is that the content is parsed before Get is evaluated so needed contexts are not present yet.

Additionally the test is run inside Block[{$Context},...] so manual Get before hitting Run won't fix that either because Block leaves only System` and Global` visible during content parsing.

---

• General solution

  Rule of thumb

  First of all, you should think about running a Test Notebook as an evaluation of a single expression, as opposed to evaluation of a list of cells in a notebook.

  It is so because the whole test notebook is parsed at once before the test is evaluated. Would be nice to have that mentioned in documentation as this makes a major difference.

  Setup

  So we need at least two test notebooks.

  • package loading test
  • functions testing notebook|s

  
  And to make it work we have to be sure that after running the first one we will have MyPackage` available on $ContextPath so that the "whatever test" notebook contents will be parsed with knowledge about MyPackage` symbols.

  To achieve that we need to run the "load test" outside the Block[{$ContextPath}... which is what Run button currently does. We also need to strip it from the latter test.

  Here is a simple function which will fix that if you evaluate TestNotebookUpdate[] in both those notebooks:

TestNotebookUpdate[] :=TestNotebookUpdate[EvaluationNotebook[]];

TestNotebookUpdate[nb_NotebookObject] := 
  (
    CurrentValue[nb, DockedCells] = Replace[
        CurrentValue[nb, DockedCells]
      , ButtonBox[
            lbl_?(Not@*FreeQ["Run-Label"])
          , opts1___
          , ButtonFunction :> _
          , opts2___
        ] :> ButtonBox[
            lbl /. DynamicBox[_["MUnitStrings", "Run-Label"], ___] ->   "Run_unblocked"
          , opts1
          , ButtonFunction :> (Block[{$ContextPath}
              , Needs["MUnit`"]]; MUnit`PaletteRun[InputNotebook[]]
              )
          , opts2
        ],
    Infinity
]);