What to be aware when using new-style package?

I will give a FAQ-style presentation of some things I became aware of while transitioning one of my packages to this format. Leonid's description of the new package format is required reading before looking at this FAQ!

How are contexts assigned to symbols in the package?

In traditional packages, symbols are looked up from $ContextPath, and if not found, created in $Context. It is always the current value of $ContextPath and $Context that determines what a symbol name in the package file refers to.

BeginPackage and Begin simply manipulate $ContextPath and $Context. When they are evaluated, the value of these system variables will change. End and EndPackage change it back. The process is dynamic, and happens as each line in the package file is evaluated one by one.

In new-style packages, the context of each symbol is decided before the contents of the package file are evaluated. Mathematica will scan all files in the package directory for the Package, PackageExport, PackageImport, PackageScope directives, and uses them to pre-determine the context of each symbol that appears in the package.

The package contents are evaluated only after this has happened.

Which contexts are symbols created in?

A private symbol in a file named Foo that is part of the package MyApp` will go in the context MyApp`Foo`PackagePrivate` .

A package scope symbol, declared with PackageScope, will go in MyApp`PackageScope` .

An exported symbol, declared with PackageExport, will go in MyApp` .

How do Package, PackageExport, PackageImport, PackageScope evaluate?

They do not evaluate at all. They are not expressions and hence they also cannot be terminated by a ; which is a short form for CompoundExpression.

They behave more like directives than symbols. They simply signal to the parser how to assign contexts to each symbol in the file. This assignment of contexts takes place before any evaluation is done.

This means that e.g. this is not valid:

PackageExport /@ {"symbol1", "symbol2"}

Instead we must write

PackageExport["symbol1"]
PackageExport["symbol2"]

PackageExport never evaluates as a symbol.

In what order are files loaded?

A new-style package is usually made up by multiple files, each containing Package["MyApp`"]. If the loading of one of these files is triggered with Get or Needs, all other files that belong to the MyApp` package will be loaded.

After the first file was loaded, the rest will be loaded in alphabetical order.

The first file would either have the same name as the package itself (e.g. MyApp.m for MyApp` ) or it would be explicitly loaded in Kernel/init.m.

What are valid file names?

Since file names are mapped to context names, file names must also be valid context names. This means that _ or a space cannot be used in file names. File names may not start with a digit.

It is a natural thought to try to control loading order by prepending digits to file names. But such names are not valid.

What is the value of $Context and $ContextPath during package loading?

In Mathematica 11.0 and later, the value of $Context and $ContextPath correspond to the file that is currently being loaded.

For example, if the current file is Main.m and the package name is MyApp` , then

$Context === "MyApp`Main`PackagePrivate`"
$ContextPath === {"MyApp`PackageScope`", "MyApp`", "System`"}

However, in Mathematica 10.x, $Context and $ContextPath do not change at all during loading. They retain the value they had before package loading (e.g. $Context === Global` and the usual $ContextPath).

What this means in practice is that in Mathematica 10.x, ToExpression["x"] would create the symbol x in the Global` context (or whatever was the context before package loading) instead of the package's private context.

The same is true for <*expr*> included in StringTemplates.

Finally, if the package loads another file with Get, all symbols in that file will be created in Global` , not in the package's private context.

What is the value of $InputFile and $Input when loading new-style packages?

In Mathematica 11.0 and later, $InputFile is always set to the specific file that is currently being loaded. It is different for each file that makes up the package.

In Mathematica 10.x, $InputFile is always set to the first file of the package, regardless of which file is currently being loaded.

$Input is always set to the name of the first file, in all versions between M10.0–M11.3.

Can we give a definition to Package?

Each file that is part of the package must include

Package["MyContext`"]

As explained before Package is just a directive, and does not have a definition. We might, in principle, give it a definition, and hope that it will evaluate in each package file.

This, however, does not happen. Package and related directives are removed from the file before its contents are evaluated.

I would add to this that it is undocumented for a reason, unfinished, and may break in the future.