diff --git a/src/mongo/db/extension/README.md b/src/mongo/db/extension/README.md new file mode 100644 index 00000000000..3b59048be11 --- /dev/null +++ b/src/mongo/db/extension/README.md @@ -0,0 +1,57 @@ +# MongoDB Extensions API + +This document aims to provide a high-level overview for the MongoDB Extensions API. +An extension is an ahead-of-time compiled object that is dynamically loaded into the server +to provide additional functionality. This object provides a handful of functions the server +may invoke to setup/teardown the extension and register new functionality. Each extension may be +updated independently from the server, meaning that functionality can be added or altered without +building and releasing a new version of the server. + +This is a work in progress and more sections will be added gradually. + +## Public API + +The Extensions API’s primary goal is to provide a header which specifies the data structures and +functions that extension developers must use and implement in order to fully implement an +aggregation stage as an extension. This API header is referred to as the Public API, and can be +under `mongo/db/extensions/public/api.h`. The Public API establishes the contracts and protocol +through which the host can load an extension, make function calls into an extension, and likewise, +how the extension can expect to interface with the host. The Public API will be versioned, vendored +and distributed to extension developers. It is written in C to ensure we maintain a stable ABI. + +## Host API + +While the Public API defines the building blocks for communicating and interacting between the host +and the extension, its C interface makes it difficult and unsafe for the host code (i.e C++) to +interact with it directly. + +The Host API is an adapter layer responsible for creating a safe interface for the C++ host code to +interact with the extension using the C Public API. The host does not need to be aware of any of the +C types that are introduced in the Public API. Instead, the Host API provides C++ classes and +functions which abstract away the complexity and memory ownership concerns of interfacing with the +C API. + +The Host API can be found under the `mongo/db/extension/host` directory. + +In general, every abstraction in the Public API has a respective C++ interface implemented in the +Host API which the host is expected to use. This allows us to encapsulate and control where +conversions across the API boundary between C and C++ take place, leading to more maintainable code +and minimizing the risk of programmer errors in the host code. The Host API code lives within the +C++ namespace mongo::extension::host. + +## SDK API + +The SDK API is an adapter layer that is responsible for creating a safe interface for an extension +developer to build an extension in their language of choice, and have it interact with the C Public +API. + +The Extensions API initiative will only support Rust extensions in production. The Search team will +own the Rust SDK. However, the Query team develops and maintains a C++ SDK for the purpose of +writing internal unit and integration tests. The C++ SDK API can be found under +`mongo/db/extensions/sdk` directory. + +In general, every abstraction in the Public API has a respective C++ interface implemented in the +C++ SDK API which extension developers are expected to use to build their extension. This includes +things like convenience methods, relevant base classes, etc. This allows us to encapsulate and +control where conversions across the API boundary between C and C++ take place, leading to more +maintainable code and minimizing the risk of programmer errors in extension code. diff --git a/src/mongo/db/extension/host/README.md b/src/mongo/db/extension/host/README.md new file mode 100644 index 00000000000..09511e22c63 --- /dev/null +++ b/src/mongo/db/extension/host/README.md @@ -0,0 +1,16 @@ +# Host API + +## DocumentSourceExtension + +Aggregation stages are modelled and implemented within the server as `DocumentSource` +specializations. The Host API introduces `DocumentSourceExtension`, a specialization of the +`DocumentSource` abstraction, which acts as a façade and delegates its business logic to the +Extensions API. + +All the methods offered by the `DocumentSource` interface are implemented by modelling a +comprehensive set of interfaces in the API which the extension must service. + +`DocumentSourceExtension` delegates calls to its interface to the following Public API components: + +- `MongoExtensionAggregationStageDescriptor` (Static Descriptor) +- `MongoExtensionLogicalAggregationStage` (Logical Stage) diff --git a/src/mongo/db/extension/public/README.md b/src/mongo/db/extension/public/README.md new file mode 100644 index 00000000000..de5a218a0f3 --- /dev/null +++ b/src/mongo/db/extension/public/README.md @@ -0,0 +1,101 @@ +# MongoDB Extensions Public API + +## Implementing Polymorphism Across API Boundary + +The API aims to provide flexibility to extension developers in choosing how an implementation +looks on the extension side of the API boundary. To this end, we provide an additional layer of +indirection when defining the data types that comprise this API, which allows us to hide data +members of API objects and implementation details entirely on the extension side of the API +boundary. + +We achieve this by implementing polymorphism in the C API, such that the vast majority of the +data structures that cross the API boundary only hold a single member: a pointer to a +virtual table that represents the common interface for the polymorphic type. + +For example, `APIStruct` below only has a single member, a pointer to `APIStructVTable`, which +requires that an extension implements the `foo` function, and assign it to the function pointer in +the `APIStructVTable` provided to the instantiation of an `APIStruct`. + +``` +// Note: APIStruct is not part of the API, just an example. +extern "C" { + // public API: + typedef APIStructVTable { + void (*foo)(); + } APIStructVTable; + + typedef APIStruct { + const APIStructVTable* vtable; + } APIStruct; +} +``` + +## Memory Ownership of Extension-Allocated Objects + +Function calls of this API will at times pass extension-allocated data structures across the API +boundary back to the host. + +When passing data structures across the API boundary, it is imperative that memory is deallocated +using the same mechanism by which it was originally allocated. Memory allocated by the extension +must be deallocated by the extension, and memory allocated by the host must be deallocated by the +host. + +For this reason, when allocated memory is passed across the API boundary with the intention of +transferring ownership to the caller, it must be done so via an interface that offers the +functionality required to delegate the deallocation back to the original allocation context. + +This API adopts the convention that all data structures that intend to transfer ownership to +the caller must provide a `destroy()` function pointer in their interface, as shown in the example +below: + +``` +extern C { + // Note: Destroyable is not part of the API, just an example. + typedef struct DestroyableVTable { + /** + * destroy `ptr` and free any related resources. + */ + void (*destroy)(Destroyable* ptr); + } DestroyableVTable; + + typedef struct Destroyable { + const DestroyableVTable* vtable; + } Destroyable; +} +``` + +In the Extensions API, the presence of a `destroy()` function in an interface indicates that the +type is associated with long lived memory whose ownership can be transferred across the API +boundary (i.e. from the extension to the host). It is important to note that when a function intends +to transfer ownership across the boundary, it must be explicitly stated and made clear in the +function’s documentation. + +## MongoExtensionStatus + +Exceptions must never cross beyond the extension’s API boundary. This means that extension +developers must guarantee that no exceptions escape from the extension, and any such exceptions +must be converted to errors that can be passed across the boundary and interpreted by the host. + +For the most part, this API adopts a convention that all function calls across the API boundary must +return a `MongoExtensionStatus` which will inform the caller whether the API call was successful or +not. A zero error code indicates success, while a non-zero error code indicates an error during +the function execution. `MongoExtensionStatus` is a long-lived allocated object, since it needs to +to provide additional error information in the failure case. + +Note that when a `MongoExtensionStatus` is returned by a function call, ownership is always +transferred to the caller of the function. Once the error is no longer needed by the caller, +its deallocation must be delegated to the other side of the API boundary. + +## MongoExtensionAggregationStageDescriptor + +A `MongoExtensionAggregationStageDescriptor` describes features of an aggregation stage that are +not bound to the stage definition. This object functions as a factory to create a logical stage +through parsing. Note, that a `MongoExtensionAggregationStageDescriptor` is always fully owned by +the extension, and is expected to remain valid for the entire time an extension is loaded. + +## MongoExtensionLogicalAggregationStage + +A `MongoExtensionLogicalAggregationStage` describes a stage that has been parsed and bound to +instance specific context -- the stage definition and other context data from the pipeline. +These objects are suitable for pipeline optimization. Once optimization is complete they can +be used to generate objects for execution.