SERVER-104308 Provide initial documentation for Extensions API (#38601)

GitOrigin-RevId: 8ba0d0c5371482d48a9632495c0a94b1e7a4bc4a

src/mongo/db/extension/README.md

@@ -0,0 +1,57 @@
+# MongoDB Extensions API
+
+This document aims to provide a high-level overview for the MongoDB Extensions API.
+An extension is an ahead-of-time compiled object that is dynamically loaded into the server
+to provide additional functionality. This object provides a handful of functions the server
+may invoke to setup/teardown the extension and register new functionality. Each extension may be
+updated independently from the server, meaning that functionality can be added or altered without
+building and releasing a new version of the server.
+
+This is a work in progress and more sections will be added gradually.
+
+## Public API
+
+The Extensions API’s primary goal is to provide a header which specifies the data structures and
+functions that extension developers must use and implement in order to fully implement an
+aggregation stage as an extension. This API header is referred to as the Public API, and can be
+under `mongo/db/extensions/public/api.h`. The Public API establishes the contracts and protocol
+through which the host can load an extension, make function calls into an extension, and likewise,
+how the extension can expect to interface with the host. The Public API will be versioned, vendored
+and distributed to extension developers. It is written in C to ensure we maintain a stable ABI.
+
+## Host API
+
+While the Public API defines the building blocks for communicating and interacting between the host
+and the extension, its C interface makes it difficult and unsafe for the host code (i.e C++) to
+interact with it directly.
+
+The Host API is an adapter layer responsible for creating a safe interface for the C++ host code to
+interact with the extension using the C Public API. The host does not need to be aware of any of the
+C types that are introduced in the Public API. Instead, the Host API provides C++ classes and
+functions which abstract away the complexity and memory ownership concerns of interfacing with the
+C API.
+
+The Host API can be found under the `mongo/db/extension/host` directory.
+
+In general, every abstraction in the Public API has a respective C++ interface implemented in the
+Host API which the host is expected to use. This allows us to encapsulate and control where
+conversions across the API boundary between C and C++ take place, leading to more maintainable code
+and minimizing the risk of programmer errors in the host code. The Host API code lives within the
+C++ namespace mongo::extension::host.
+
+## SDK API
+
+The SDK API is an adapter layer that is responsible for creating a safe interface for an extension
+developer to build an extension in their language of choice, and have it interact with the C Public
+API.
+
+The Extensions API initiative will only support Rust extensions in production. The Search team will
+own the Rust SDK. However, the Query team develops and maintains a C++ SDK for the purpose of
+writing internal unit and integration tests. The C++ SDK API can be found under
+`mongo/db/extensions/sdk` directory.
+
+In general, every abstraction in the Public API has a respective C++ interface implemented in the
+C++ SDK API which extension developers are expected to use to build their extension. This includes
+things like convenience methods, relevant base classes, etc. This allows us to encapsulate and
+control where conversions across the API boundary between C and C++ take place, leading to more
+maintainable code and minimizing the risk of programmer errors in extension code.

src/mongo/db/extension/host/README.md

@@ -0,0 +1,16 @@
+# Host API
+
+## DocumentSourceExtension
+
+Aggregation stages are modelled and implemented within the server as `DocumentSource`
+specializations. The Host API introduces `DocumentSourceExtension`, a specialization of the
+`DocumentSource` abstraction, which acts as a façade and delegates its business logic to the
+Extensions API.
+
+All the methods offered by the `DocumentSource` interface are implemented by modelling a
+comprehensive set of interfaces in the API which the extension must service.
+
+`DocumentSourceExtension` delegates calls to its interface to the following Public API components:
+
+- `MongoExtensionAggregationStageDescriptor` (Static Descriptor)
+- `MongoExtensionLogicalAggregationStage` (Logical Stage)

src/mongo/db/extension/public/README.md

@@ -0,0 +1,101 @@
+# MongoDB Extensions Public API
+
+## Implementing Polymorphism Across API Boundary
+
+The API aims to provide flexibility to extension developers in choosing how an implementation
+looks on the extension side of the API boundary. To this end, we provide an additional layer of
+indirection when defining the data types that comprise this API, which allows us to hide data
+members of API objects and implementation details entirely on the extension side of the API
+boundary.
+
+We achieve this by implementing polymorphism in the C API, such that the vast majority of the
+data structures that cross the API boundary only hold a single member: a pointer to a
+virtual table that represents the common interface for the polymorphic type.
+
+For example, `APIStruct` below only has a single member, a pointer to `APIStructVTable`, which
+requires that an extension implements the `foo` function, and assign it to the function pointer in
+the `APIStructVTable` provided to the instantiation of an `APIStruct`.
+
+```
+// Note: APIStruct is not part of the API, just an example.
+extern "C" {
+    // public API:
+    typedef APIStructVTable {
+        void (*foo)();
+    } APIStructVTable;
+
+    typedef APIStruct {
+        const APIStructVTable* vtable;
+    } APIStruct;
+}
+```
+
+## Memory Ownership of Extension-Allocated Objects
+
+Function calls of this API will at times pass extension-allocated data structures across the API
+boundary back to the host.
+
+When passing data structures across the API boundary, it is imperative that memory is deallocated
+using the same mechanism by which it was originally allocated. Memory allocated by the extension
+must be deallocated by the extension, and memory allocated by the host must be deallocated by the
+host.
+
+For this reason, when allocated memory is passed across the API boundary with the intention of
+transferring ownership to the caller, it must be done so via an interface that offers the
+functionality required to delegate the deallocation back to the original allocation context.
+
+This API adopts the convention that all data structures that intend to transfer ownership to
+the caller must provide a `destroy()` function pointer in their interface, as shown in the example
+below:
+
+```
+extern C {
+   // Note: Destroyable is not part of the API, just an example.
+   typedef struct DestroyableVTable {
+        /**
+         * destroy `ptr` and free any related resources.
+        */
+        void (*destroy)(Destroyable* ptr);
+   } DestroyableVTable;
+
+   typedef struct Destroyable {
+        const DestroyableVTable* vtable;
+   } Destroyable;
+}
+```
+
+In the Extensions API, the presence of a `destroy()` function in an interface indicates that the
+type is associated with long lived memory whose ownership can be transferred across the API
+boundary (i.e. from the extension to the host). It is important to note that when a function intends
+to transfer ownership across the boundary, it must be explicitly stated and made clear in the
+function’s documentation.
+
+## MongoExtensionStatus
+
+Exceptions must never cross beyond the extension’s API boundary. This means that extension
+developers must guarantee that no exceptions escape from the extension, and any such exceptions
+must be converted to errors that can be passed across the boundary and interpreted by the host.
+
+For the most part, this API adopts a convention that all function calls across the API boundary must
+return a `MongoExtensionStatus` which will inform the caller whether the API call was successful or
+not. A zero error code indicates success, while a non-zero error code indicates an error during
+the function execution. `MongoExtensionStatus` is a long-lived allocated object, since it needs to
+to provide additional error information in the failure case.
+
+Note that when a `MongoExtensionStatus` is returned by a function call, ownership is always
+transferred to the caller of the function. Once the error is no longer needed by the caller,
+its deallocation must be delegated to the other side of the API boundary.
+
+## MongoExtensionAggregationStageDescriptor
+
+A `MongoExtensionAggregationStageDescriptor` describes features of an aggregation stage that are
+not bound to the stage definition. This object functions as a factory to create a logical stage
+through parsing. Note, that a `MongoExtensionAggregationStageDescriptor` is always fully owned by
+the extension, and is expected to remain valid for the entire time an extension is loaded.
+
+## MongoExtensionLogicalAggregationStage
+
+A `MongoExtensionLogicalAggregationStage` describes a stage that has been parsed and bound to
+instance specific context -- the stage definition and other context data from the pipeline.
+These objects are suitable for pipeline optimization. Once optimization is complete they can
+be used to generate objects for execution.