Engineering

Enhancing Countly with Dynamic Plugin Enabling and Disabling

Last updateD on
February 20, 2024
Countly's Dynamic Plugin Enabling and Disabling

For almost a decade now, Countly has been extendable through the plugin system. But it also had its cost. In this article, we will explore how it worked before, what the limitations were, and what we did to improve the situation.

Background in Countly plugin system

Unlike other languages (like PHP, which dynamically loads all files on each request), NodeJS loads the files once per process. And it imposes multiple restrictions when making it extendable by plugins that can be enabled or disabled at any time (not even including the development part of those plugins).

Since we wanted to get Countly off the ground with the plugin system as soon as possible, we opted for a more crude but easier approach. When you toggle any plugin, we would simply restart the NodeJS process and regenerate the whole frontend minified files again. The idea was that we would not do it that often anyway. And as usual, we were wrong.

However, multiple problems arose. Firstly, lots of people started using them as feature toggle, disabling if there were any performance or usage issues. 

Then as the front end grew bigger and more complicated, regenerating became slower and slower. With the introduction of sass, it became unbearably slow. 

There are also problems with browser-side caching, as it all needs to be updated on each regeneration. 

And the straw that broke the camel's back was docker images, where the user could have any dynamic plugin set provided on docker image start, and as we did not know if any plugin was already installed, we had to reinstall all the plugins on every image start and regenerate all frontend minified files, which proved to be a very long booting time for docker images.

Eventually, we decided enough was enough, and something had to change. And that is what we did with our 23.03 release.

New dynamic plugin system approach

Since it is not possible to unrequire files in running the NodeJS process (well, it is possible, but it is not a clean thing to do), so with that out of the way, we knew we needed to disable the plugin while preserving everything as is (including minified frontend files). So we identified 5 main pain points in our architecture that prevent us from doing dynamic plugin toggling, and we tackled them one by one. 

Saving plugin states

In the old approach, the plugin state was saved in the file. All servers/and docker images running on the same deployment would need to have the same configuration of the plugin file, which was manageable, since there are also other configurations, like connection to the database, etc.

But with a dynamic approach, we needed a way to share the plugin states across multiple servers in the same deployment because they could change run time. For that purpose, we added a plugin section in our shared config document in countly.plugins collection. 

Now if the plugin is not installed, it would not appear in that document, and core would automatically run installation scripts for it. If the plugin is enabled, it would have a property with the plugin name as key and value true, indicating that. And, of course, if it is disabled, the value would be false. And all places checking for plugin states would reference that document.

Here is the code doing the initialization check:

https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L166-L180

API backend

On the Countly API backend, all plugins communicate through event-like systems. They register and dispatch events through the Countly plugin manager. 

If the plugin is enabled, it is not a problem, and it will subscribe and receive events. But if the plugin is disabled, then it can subscribe to the events on process start but should not receive any event until it is enabled.

And as you can imagine, it was relatively easy to do so. We would know which plugin is listening and simply not propagate even to that plugin (the perks of going with our own event system, rather than using NodeJS built in one - more control).

Here is the code that does it in our plugin manager: 

https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L649-L652

Dashboard backend

Another part that communicates with all the plugins is the backend for dashboards. Since it does not experience the heavy load that API is subjected to, we used express js there from the start, and adding plugins was just adding more middleware - which was easy. Changing this now to disabling middleware, unfortunately, was not easy.

The first thought was to add a check in each and every plugin if it should process the middleware or skip it. But it would mean modifying every single plugin, and we wanted to make it as backward compatible as possible, so this approach would not work for us.

The second way was to mangle with express js internals to skip middlewares automatically from the inside, but it would potentially break if express js changed something internally, so it was deemed to be too risky.

The approach we went with was actually to create a layer upon express js, which allowed us to wrap each added middleware in all our middleware, which would check if the plugin's middleware should be processed or not. It may not sound like an entirely clean approach, but it works and would not need any change on the plugin sides, and it would only break if Express JS changes its interface with middleware.

Here is the code where we wrap plugin middlewares in our own before passing them to express js:

https://github.com/Countly/countly-server/blob/23.03/plugins/pluginManager.js#L755-L826

Countly Jobs

Next Countly plugin interface is the background jobs that plugins can schedule and run code periodically. But since we already had a central place that managed all jobs, it was easy just to add checks before jobs were run.

Here is the code doing exactly that:

https://github.com/Countly/countly-server/blob/23.03/api/parts/jobs/manager.js#L265-L268

Browser side

The last part, which is also the only one that is less backward compatible, is the browser part, which also has a state of minified files combined together. So for all plugins, there is only one single minified javascript file. 

Since the minification process was very long and resource-consuming, we had to go at it from the perspective that minified files should not change. We just need to enable/disable plugin functionality there.

And there are three parts to that:

  • Adding plugins to the menu -  there is a central place to do that, but we don't know which plugin actually registers the menu, so if your plugin name (the folder name) matches your permission name, then it is done automatically. If not, you need to provide pluginName property to the menu object.
  • Adding plugins to the routes, we decided not to block that because the backend is already disabled, and someone can get to the plugin only if they go directly to that URL, and it would not be functioning properly either way.
  • Globally scoped plugin functionality - this is the only thing that is not backward compatible and needs to be added to specific plugin checks.

Join our growing community on GitHub

Join our growing community on GitHub

Join our growing community on GitHub

Join our growing community on GitHub

Follow Countly

How to make sure your plugins are compatible with dynamic plugin toggling?

The only backward incompatible part is the browser side, and there are only two things you need to check.

The first one is adding a plugin menu/submenu/tab/etc. If your plugin permission matches your plugin name (plugin folder name), then all good, you don't have to do anything. But if they do not match, you will need to provide the pluginName property like this:

https://github.com/Countly/countly-server/blob/23.03/plugins/compliance-hub/frontend/public/javascripts/countly.views.js#L661

And the other part is to make sure you scope your plugin and perform operations that are outside of the plugin view (like adding links and parts for other plugins/sections) after checking if the plugin is enabled. Here is the code example of that:

https://github.com/Countly/countly-server/blob/23.03/plugins/locale/frontend/public/javascripts/countly.models.js#L160-L162

Conclusion

Now, enabling and disabling plugins in Countly does not require any process restart or changes to the file structure.

This has been a big change with the best possible backward compatibility we could have. But the value of this is not obvious right away. But soon, more articles will come out that rely on this new feature.

Stay tuned to our blog and join our Discord community server to stay up-to-date with the latest technical product updates:

Thank you! The Data Privacy Checklist will be in your inbox shortly.
Oops! Something went wrong while submitting the form.
TAGS
Countly
Product Development

Subscribe to 🗞️
our newsletter

Join 10,000+ of your peers and receive top-notch data-related content right in your inbox.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Get started with Countly today 🚀

Elevate your user experience with Countly’s intuitive analytics solution.
Book your demo

Get started with Countly today 🚀

Elevate your user experience with Countly’s intuitive analytics solution.
Book your demo

Try Countly Flex today

Privacy-conscious, budget-friendly, and private SaaS. Your journey towards a product-dream come true begins here.

Posts that our readers love