mongo/docs/modularity.md

# Modules API

## What is a module

A module:

- Provides a coherent public API
- Has internal details that are not intended to be directly accessed from outside the module
- Is a set of files covering the API (headers), implementations (headers and cpp files), and tests

### Submodules

TODO

## Why are we doing this?

Having a clear delineation between public and private APIs for each module will improve the
maintainability and velocity of our codebase. Teams will have more freedom to evolve their internal
implementation details without affecting consumers. Consumers will benefit from knowing what APIs
are intended for their consumption.

## Assigning files to modules

The file `modules_poc/modules.yaml` contains a list of modules, each containing a list of files.
Each file must be contained in only one module. Note that module assignment is not required to map
neatly to team ownership.

In cases where multiple globs match a file, the current rule is that the longest glob wins. This is
used as a simpler-to-implement version of most-specific glob wins, which we may switch to in the
future.

## How do I mark API visibility?

This section will just describe the basic process. Later sections will cover the tooling available
to help, along with caveats to be aware of.

First read the documentation in
[src/mongo/util/modules.h](https://github.com/mongodb/mongo/blob/master/src/mongo/util/modules.h) for
the canonical list and description of visibility levels. As a brief overview of the main levels from
least to most restrictive:

- `OPEN`: This is available for usage _and inheritance_ from anywhere in the codebase
- `PUBLIC`: This is available for usage from anywhere in the codebase. For types, subclasses may
  only be defined in the same module.
- `NEEDS_REPLACEMENT` and `USE_REPLACEMENT(...)`: These are collectively considered "unfortunately
  public" and are available for use, but should be avoided
- `PARENT_PRIVATE`: This is similar to `PRIVATE`, but allows usage from any file in the parent
  module, including other submodules
- `PRIVATE`: This may only be used from the current module or one of its submodules
- `FILE_PRIVATE`: This may only be used from the current "file family" (roughly, header \+ cpp \+
  tests). It may not be used by other files, even from the same module.

You can think of public vs private similarly to how you would the sections of a `class`: they
indicate whether something is intended to be part of the API or an implementation detail. The
difference is that they apply at a wider granularity of code than a single class, with
implementation details available to either the full module (and its submodules) for `PRIVATE` or the
file family for `FILE_PRIVATE`.

The macros in that header file are attached to declarations and set the visibility level for that
declaration and all of its "semantic children"[^1]. The macros are C++ attributes which means that
they need to go in specific places that differ based on what is being marked (for templates, the
location does not change and is always somewhere after the `template <...>` part):

- `MONGO_MOD_PUBLIC;` by itself as the first line after includes in a header sets the default for
  that header (only `PUBLIC`, `PARENT_PRIVATE`, and `FILE_PRIVATE` are allowed here)
- `namespace MONGO_MOD mongo {` (this does not work with nested namespaces in a single declaration
  like `namespace mongo::repl`)
- `class MONGO_MOD Foo {` (Ditto for `enum`, `struct`, and `union`)
- `MONGO_MOD void func(...);`
- `MONGO_MOD int var;`
- `concept isFooable MONGO_MOD {`

For the cases where it goes at the beginning of the line, if clang-format chooses an unfortunate
place to break the line, it usually helps to undo the formatting then put the macro on its own line
above the declaration.

APIs are marked one header at a time, by including `"mongo/util/modules.h"` in the header. This
causes the header to be treated as "modularized" which has the following effects:

- All declarations in that header (not transitive includes) default to `PRIVATE`, meaning that the
  public API is what must be marked.
- Members in `private:` sections in classes default to `PRIVATE`, regardless of the visibility of
  the class. The only way the language would allow them to be used from outside of the module is if
  you have cross-module friendships, which should generally be avoided. If needed temporarily, favor
  `NEEDS_REPLACEMENT` over `PUBLIC` for these declarations.
- Declarations ending in `_forTest` default to `FILE_PRIVATE` to support the common case where they
  are only intended for testing that class. If they are actually intended to support testing of
  consumers, not just the type they are defined on, they can be explicitly given `PUBLIC` or
  `PRIVATE` visibility.
- Internal and detail namespaces default to `PRIVATE` and cannot be made less restricted, but can
  still be marked as `FILE_PRIVATE`. Individual declarations within the namespace can be exposed as
  necessary, but they cannot be exposed in bulk without changing the name of the namespace to
  something that doesn't imply private.

For internal headers of a module which do not contribute to its public API, simply including
`modules.h` is sufficient. There is a [tool](#the-private-header-marker) to automate this process.
You may additionally want to consider whether any APIs should be marked `FILE_PRIVATE`, but that is
optional.

For IDL files, you mark visibility of whole types (`struct`, `enum`, and `command`) with the
`mod_visibility` option. The value should be the same as one of the `MONGO_MOD` macros, but
lowercase and without the prefix, for example `mod_visibility: public`. You can set the default
visibility for all types in that IDL file by putting that in the `global:` section. You cannot
control visibility of individual functions within the type. Please let us know if you have a
compelling use case for this.

## What tooling exists to help me?

Note that all tooling should be run from within a properly set-up python virtual environment. This
includes running `buildscripts/poetry_sync.sh` to ensure you have the correct dependencies.

### The scanner and merger

The merger generates a cross reference of all first-party usages of first-party code and stores it
in `merged_decls.json`, which is used by the rest of our tooling. It is also where we validate that
there are no disallowed accesses. It will be invoked for you by the browser when you ask it to
rescan, or you can also manually run it as `modules_poc/merge_decls.py`. If you are interested in
analyzing that file, [`jq`](https://jqlang.org/) is a powerful tool, or you can just write some
python.

As a rather extreme example of what you can do with `jq`, here is how the progress reports are
generated:

```shell
# For each mod (and TOTAL):
#   For each file:
#     consider it marked if it has no UNKNOWNs
#   Compute a done percentage
#   Format to a nice string
jq 'map(., .mod = "TOTAL") | group_by(.mod)[] | group_by(.loc | split(":")[0]) | {mod: .[0].[0].mod, total: length,  marked: map(select(any(.visibility == "UNKNOWN") | not)) | length} | .done = (1000 * .marked / .total | round) / 10 | "\(.mod): \(" " * (.mod | 40-length)) \(.done)%  (\(.marked) / \(.total))"' -r merged_decls.json
```

Internally, the merger will internally invoke `bazel build --config=mod-scanner //src/mongo/...` to
run the scanner over the whole codebase (or the parts that have changed since the last scan), taking
advantage of bazel remote execution to achieve very high levels of parallelism.

### The browser

The main piece of tooling to run is the browser, which is launched by running
`modules_poc/browse.py`. If you haven't scanned the codebase recently, it will offer to run it for
you which will take a few minutes. After modifying the source code, you can rescan at any time by
pressing `r`. It will only rescan files that have been modified or that transitively include
modified headers.

The browser is primarily intended to assist in labeling public APIs, so the files are sorted with
the most number of unlabeled declarations ("unknowns") first. You can search for a file by pressing
`f` or press `m` to filter the files by module.

The list of available key bindings is shown on the right. You can toggle that by pressing `?`. Other
keybinding of note are that you can press `g` to go to the currently highlighted declaration or
location in your editor (only when running in the vscode or nvim terminal), and `p` to toggle an
inline preview of the location within the browser. You can press `Tab ↹` to toggle between the tree
and the code preview. The mouse is fully supported for scrolling and expanding rows in the tree, and
there are aliases for some basic vim keybinds (`hjkl/`).

### The private header marker

Once you have scanned the codebase and produced a `merged_decls.json`,
`modules_poc/private_headers.py` can be used to find all header and IDL files where there are no
currently detected external usages and automatically mark them as fully private to the module. This
does not necessarily mean that all automatically marked headers are intended to be private. A human
should review to ensure that the marked headers match intent. You can pass flags to filter on
any/all of module, owning team, or path glob. For headers matching the filter, the script will also
warn of usages of `_forTest` external to the file family that may need to be marked `PRIVATE` to
make them available to the whole module since they default to only being available to the file
family for marked headers.

Make sure to run `buildscripts/clang_format.py format-my` or `bazel run format` after using it to
modify any C++ files.

Example usage:

```shell
./modules_poc/private_headers.py --team=server_programmability --module=core --glob="src/mongo/executor/*"
```

`--dry-run` can be added to view all of the changes without applying them.

### The PR comment generator

You can run `modules_poc/mod_diff.py` to output a brief summary of all of the API (including
visibility levels and usages counts) for each file modified in your branch. When putting up a PR to
mark API visibility, you should add a comment with its output to the PR as an aide to reviewers. The
output is intended to be close enough to C++ that you should put it in a ` ```cpp ` block when
making your PR comment to make it more readable. You can also pipe it through `bat -lcpp` to make it
colorful locally. Note that it will use the last scan output, so if you've modified any headers, you
should run a rescan prior to running this tool.

## Workflow

The general workflow for each PR will generally be the same:

1. Ensure that you are in a python virtualenv, creating one if needed, and run
   `buildscripts/poetry_sync.sh` to update python deps.
2. Run [the merger](#the-scanner-and-merger) to scan the code base: `modules_poc/merge_decls.py`
3. Mark some headers
4. Rerun the merger to ensure that there are no violations, and update `merged_decls.json`
5. Run [the pr comment generator](#the-pr-comment-generator) to show the APIs that you have marked
   - Look through this to ensure that everything is as you expect.
6. Put up a PR and include the generated comment in a ` ```cpp ` block
   - I suggest keeping PRs small (say, no more than 10 files at a time) so that they are manageable
     by reviewers. As an exception it seems reasonable to auto-mark many headers as private in a
     single PR, as long as those PRs are separate from those containing any manual marking.

When first starting to mark a module, I suggest running the
[`modules_poc/private_headers.py`](#the-private-header-marker) script with `--dry-run` (or `-n`) and
`--module=YOUR_MODULE`. For larger modules (in particular, the `query` mega module) you may want to
pass a `--glob` so that you can focus on a smaller subset of the code initially. That will give you
an overview of the files that are used from outside your module (which contain defacto public APIs
today) and those that do not (which can automatically be marked as private implementation details).

If all of the defacto private headers seem like they should be private, you can remove the dry-run
flag to have it automatically mark them as private. Be sure to validate that their contents are
actually intended to be private. Remember that the point of having a human doing the marking is to
ensure that we correctly capture intent. You can optionally mark implementation details within each
header as `FILE_PRIVATE`, if you would like to prevent them from being used elsewhere even within
the module.

You can then open [the browser](#the-browser) (`modules_poc/browse.py`) to look at the remaining
headers. It will show you what is used and from where. It will be particularly useful for things
that seem like they should be private, but are being used externally.

### What should I do when an internal API is currently being used?

1. If it is only used from a small number of external files, first check if those files should
   actually belong to your module. We tried to correctly map all files in phase 1, but some files
   may have been assigned to the wrong module. If that happens, try adjusting the globs in
   `modules_poc/modules.yaml` to move them.
2. If there is already a public API that callers should use instead, mark it as
   `USE_REPLACEMENT(better_api)`. The argument accepts any C++ tokens, but the intent is where
   possible to use the name of the replacement. This will generate a ticket for all teams using that
   code.
   1. If there are very few users, consider just cleaning them up.
3. Reconsider making this API public if other modules need its functionality, and this is the only
   way to get it.
4. Otherwise, if there is no public API that fulfills the needs of the callers, but you don't want
   the current API to remain public long-term, use `NEEDS_REPLACEMENT`. This will generate a ticket
   for the team that owns that code.
   1. If the API was "obviously" intended to be private (eg it is in a `details` namespace) and
      callers would be reasonably able to implement the functionality themselves, possibly by
      writing their own version, it seems acceptable to use
      `USE_REPLACEMENT(do not use internal details)`

## Caveats and Limitations

**OVERARCHING GUIDELINE**: Always try to mark declarations correctly according to intent, even if it
will not be enforced by the current tooling. This is both to provide the correct information to
human readers, as well as to avoid issues if we improve the tooling in the future to eliminate these
limitations

The rest of this section is fairly technical and probably not necessary for most readers unless they
notice something "weird" going on and want to dive into why. Most of these limitations are more
likely to affect the core modules since most of the rest of our code does not expose APIs via macros
and templates or have APIs only consumed by templates, and those are where most of these issues come
up.

- We do not track usages of namespaces at all, only the declarations within namespaces. When a
  namespace is marked with a visibility, it does not affect the visibility of the namespace itself
  (since it doesn't have one), it sets the default visibility for all declarations within **that
  namespace block**. Each time a namespace is reopened it is a separate block and the visibility
  markers on other blocks of the same namespace do not apply.
- The scanner only knows about declarations that it sees being used. For implementation reasons, it
  only discovers declarations by seeing what every usage is using. This can either cause or be
  caused by other limitations.
- Usages in templates may not be seen. This is especially the case for "dependent types and values"
  which are things that are not known by the compiler before the template is instantiated.
  - This is a problem for functions where any arguments are dependent if it can't figure out which
    overload will be selected. It is even worse for free-functions called unqualified (`f(blah)`
    rather than `ns::f(blah)` or `x.f(blah)`) since due to ADL, overload resolution is _always_
    delayed for them.
- Everything that results from a macro expansion is treated as-if it was written at the point of
  expansion. This applies to both declarations and usages. If you have an API that should only be
  used via the defined macros, mark it as `MOD_PUBLIC_FOR_TECHNICAL_REASONS` to signal to readers
  that they should avoid direct usage, even if the tooling won't prevent it. We may improve this in
  the future.
- Template variables are completely ignored due to some unfortunate clang bugs. Still, try to mark
  them correctly since we may change this in the future.
- Method calls are assigned to the static type at the call site. This has two important effects:
  - A subclass's overridden method may seem unused if it is only used via calls through a base class
    pointer/reference
  - Calls through a base class pointer/reference count as calls of that class's method, not of the
    interface's
- Defaulted members (methods, ctors, dtors) are treated as usages of the class itself, regardless of
  whether they implicitly or explicitly defaulted. This is because clang does not provide an API to
  distinguish between those cases.
- Template normalization woes: we try really hard to report declarations as the template `foo<T>`
  rather than separate instantiations like `foo<int>`, `foo<string>`, etc, **unless** they are
  explicitly specialized, meaning that the instantiation has its own definition different from the
  main template. Unfortunately, clang does a bad job at this and we have a number of kludgy
  workarounds. The most important effects:
  - Explicit specializations of function and variable templates are ignored and always converted to
    the primary template.
  - We do treat explicit specializations of types as separate (using the heuristic of having a
    separate location than the main template), because they can have a different shape and API than
    the main template. In general they should probably have the same visibility though, unless the
    instantiation is using a private type which should be unavailable to consumers anyway.
  - Clang assigns many locations to the site of explicit template instantiations and extern template
    declarations, even when there is a better location that it can see. Luckily these are fairly
    rare.
- Sometimes clang reports the resolved destination of `using` declarations and type alias, but
  usually it reports the `using` declaration itself. A few notable cases (these are trends and may
  not be absolute\!)
  - `using Base::foo;` to expose a member of a base class is resolved as a usage of `Base::foo`
    rather than `Derived::foo`. This is especially notable when the `Base` class is intended to be a
    private implementation detail. You will need to mark all exposed methods as public.
  - `using Base::Base;` to pull in the base constructors is the opposite and is recorded as a usage
    of `Derived::Base(args)`, which is odd because such a declaration doesn't actually exist.
- Internal/details namespaces (currently defined as matching the regex `(detail|internal)s?$`)
  implicitly have implicit default visibility of private if `modules.h` is included. It is not
  possible to give the namespace a public visibility, but you can restrict it further with
  `FILE_PRIVATE`. If you want declarations inside it to be usable from outside your module you must
  mark children of the namespace explicitly, or rename it to not use a name that implies that it is
  for internal usage only. A somewhat common case will be marking internal declarations that are
  only intended to be used via macros with `PUBLIC_FOR_TECHNICAL_REASONS`.
- Be very careful with forward declarations. Try to avoid them wherever possible (unless there is a
  significant benefit). Especially avoid forward declaring anything from another module\! Where
  forward declarations must be used, make sure that they have the same visibility as the real
  definition. As an exception, if every TU that sees the forward declaration will also see the
  definition it is OK to omit marking the forward definition. This may happen when they are both in
  the same header, or the forward declaration is in a private implementation detail header which is
  included by the defining header. Be aware of the implicit visibility marking which also applies to
  forward declaration, if they are the only declaration seen in the TU.
  - Never forward declare functions to avoid including a header. They are much more problematic than
    types, both in general in C++ and specifically for this tooling.
- We try to use the definition location for types defined in headers, but the "canonical" location
  (clang's term for the first declaration seen in the current TU) for everything else. If the type
  is defined in a .cpp, we use the canonical location.
- We only consider declarations in headers, never in .cpp files.
- Be mindful of `_forTest` functions. They default to `FILE_PRIVATE` since they are typically
  intended only for use when testing the type they are defined on, not when testing consumers. In
  the cases where they _are_ intended as part of the API for testing consumers, you can explicitly
  mark them `PUBLIC` or `PRIVATE` depending on whether they should be usable from outside your
  module or not.
- Things used implicitly (eg implicit conversion operators) are still counted as usages even if they
  are not specifically named at the call site
- When merging information from multiple TUs, definitions always replace the metadata gathered from
  TUs that only saw a declaration.
  - Note that we aren't guaranteed to see every definition, in particular for functions that are not
    called from the TU that they are defined in. So this cannot be used to find places where we
    deleted the definition but forgot to delete the declaration (we wouldn't see them anyway, since
    we only track things that are used, and undefined things can't really be used, except trivially,
    without breaking the build).
- `private` members of classes are implicitly `PRIVATE`, and must be explicitly marked otherwise if
  desired. They should probably never be made `PUBLIC` since that implies cross-module friendship.
  In the few places where we have that today, they have been made one of the flavors of
  unfortunately public: `NEEDS_REPLACEMENT` or `USE_INSTEAD`.
  - `public` members of `private` types do not inherit the implicit `PRIVATE` and follow the normal
    rule of looking for their nearest semantic parent with an explicit marker. That means that they
    may be `PUBLIC`. However, the language rules still apply and as long as an instance of the type
    is never handed to consumers they will have no way of accessing those members.
  - `protected` members do not default to `PRIVATE`, but because we only allow subclassing from
    `OPEN` classes, the language visibility rules will disallow access from outside the module
    unless you choose to allow it by use `OPEN` classes or `friend`s. Note that making any subclass
    `OPEN` exposes all `protected` members of parents unless they are marked `PRIVATE`.
- `friend` declarations are mostly ignored, except when they are a definition. So the definitions
  using the "hidden friend" pattern are tracked, but we ignore it if the definition is in a cpp
  file.

[^1]:
    Clang distinguishes between "semantic" and "lexical" parents. The primary differences are that
    members of classes (including member types) are semantic children of the class even when defined
    out of line, and conversely `friend` declarations are not, and instead are considered semantic
    children of the nearest namespace.