I am working on a game engine in C++. One of the design decisions I made was to separate a “public” and “private” API, with the former being visible to clients, while the latter is used internally by the engine. The idea is to both limit the client API to whatever I want them to be able to use (and thus not be distracted by engine internals), and to hide the specific implementation modules used by the engine, e.g platform-specific code and libraries (DirectX, SDL, etc.)

Within this framework, I also want to set up a system for managing various engine resources. The engine has ownership of the objects it creates, since these objects may reference resources that the engine does not expose to the client (and must be cleaned up once the client is done with the object). So for creation and destruction, the client must call into engine systems, and then be provided with a way to access the relevant objects. A trivial implementation would be something like this:

// Visible to client:
namespace PublicAPI
{
    class Object
    {
    public:
        virtual void foo() = 0;
    protected:
        int m_id = -1;
    };

    class Manager
    {
    public:
        virtual int add_object() = 0;
        virtual Object* get_object(int object_id) const = 0;
        virtual void remove_object(int object_id) = 0;
    };
}

// Only in backend:
namespace PrivateAPI
{
    class Object final : public PublicAPI::Object
    {
    public:
        void foo() override;
    };
    
    class Manager final : public PublicAPI::Manager
    {
    public:
        int add_object() override;
        Object* get_object(int object_id) const override;
        void remove_object(int object_id) override;
    };
}

// Example usage:
PublicAPI::Manager& manager = get_manager(); // This will point to an instance of PrivateAPI::Manager
PublicAPI::Object* object = manager.get_object(object_id);
if(object)
{
	object->foo();
}

NOTE: the OOP may be misleading, at runtime each interface will only have a single implementation each. The abstraction is purely to separate public and private code.

A specific example would be if I'm trying to wrap the specific API of render window management. In DirectX, this means having a swap chain and a render target view, for which I would implement an object and a manager in the PrivateAPI. The client doesn't need to know about DirectX being under the hood, all it needs is a public Window interface which they can resize, direct render tasks to it, and so on.

In general, the idea is to hide the data within the objects (which only the PrivateAPI should know about) and only provide an interface to the functionality, which is intended to be agnostic to the implementation.

The above approach would work, but one serious drawback is that it exposes a pointer to the object directly, meaning the user could technically call delete on it, instead of doing it through the manager. There are ways to manage this, such as runtime errors or special wrapper objects that do not allow destroying the object itself, but I am not sure if it's actually worth the effort, instead of just expecting the users to understand that they shouldn't be doing it (SDL and DirectX seem to agree). Another drawback is that, under the hood, Object might need a lot of extra dependencies to access the rest of the engine and actually perform the tasks, whereas making Object into a POD class might be more efficient (referring back to the render window example: it would mean Window now needs access to the D3D device and context at the very least, just to get anything done)

Another approach would be to not provide pointers at all, and instead move all the functionality of Object into Manager, with each function taking an ID to identify which object to manipulate (i.e Manager::foo(int object_id)). This solves most of the above issues, but now the API becomes a bit of a mess. It's much harder to nicely encapsulate object-specific functionality, each manager would just have a ton of functions crammed into them, not to mention every operation would require a lookup step. This can be somewhat mitigated in various ways, but again it feels like it might be an unnecessary compromise.

With all that said, is there an elegant design pattern which fits this setup? Or is the design itself flawed? What are good case studies of other engines solving this problem?

Since I never make any public API to real users using C++, I'll give opinion from me as a user point of view.

iirc, Unreal exposes the naked pointer for many game objects but with explicit contract to the users that they should only use them, and not manage the pointer themselves, as the game objects are handled by their GC. Android's C NDK always tell when and when not to destroy the stuff provided by them. If you ask me, I would not manage any naked pointer (or any other form of managed object especially with “get” in front of the function name) served by the API unless being explicitly told to do so, just like the two examples above. Although the two examples mentioned are in two different software layers, I think the target users determines your API design.

So depending on your target users, you may or may not expose the pointer for various reasons. Maybe your target users are people who barely care/sloppy about their memory, and you wish them to pass only data that your API will handle it, like “magic”. This is likely easy to use, but limited to power users. Maybe your target users are power users, where you give access to your engine so that they can do things themselves with your tool, maybe at best including ability to pass their own memory they managed (like Vulkan VkAllocationCallbacks for the CPU side and (almost) completely manual GPU memory management unlike OpenGL) if you let them to go as low as that. Maybe you want to cater both, meaning you can create another abstraction layer that can adapt both design without messing with your low level layer codes. For example your code above can be completely separate to your internal engine, so you can do another API sample in different design, as if your API(s) itself is the user of the internal layers.

I personally would allow a pointer being exposed as long as it's consistent throughout your design and explicit on the doc on how the user is expected to do with it. Again, it's up to your target users (includes you yourself, if this is only for you).

@mychii Thanks for the reply! I've had some discussions with other devs since I posted, and their feedback pointed mainly toward the former option (i.e users only get limited access via an ID). It's probably the most viable approach overall to grant a “basic API” that covers most use-cases, and I can always expand that with more low-level access like you proposed.

Another option is to create a wrapper class that hides the pointer and exposes the API, ie

class XWrap {
private:
   X *xp;
public:
   XWrap(X * xp) { this.xp = xp; }
   int doSomething(int data) { return this.xp->doSomething(data); }
}

A user can have an instance of this class without the ability to delete *xp.

Compiler will likely eliminate the wrapper if you turn on optimization.