Recently we introduced one breaking change in how user information is exported from Countly, and we wanted to explain why such change was made and what to do to keep your plugins supported. But before we dive into why's, let's first reiterate the "what" part.
In Countly, data is stored in multiple different collections. Usually, plugins can create their own collections, connect to the data stream and process needed data themselves, and then store it in their own collections. In some cases, such data is aggregated, and it does not identify any user in particular. In other cases, that data needs to be tied to a specific user, in which case, our suggested method is to tie data using uid value of the user.
Here is an example of a crashes plugin using uid value to tie specific crashes to app users and then marking those crashes as resolved for those users on new app versions:
And if you store such data that is tied to a user, there are also other things that you must do:
In this case, we are interested in the latter, and the reason for needing such functionality is mostly compliance with regulations. Many regulations like GDPR require you to have the ability to export and provide to the user all the information that you have on them.
Currently, this is handled by Countly core itself, and all that plugin has to do is to listen to "/i/app_users/export" and provide an array of mongoexport commands that need to be run to export the user data from the plugin.
Here is again the example from the crashes plugin:
What Countly core then does is run all the provided export commands and export all the data and then provide it in UI for the dashboard user to retrieve. But there are multiple problems with this approach:
Due to these facts and other performance and concurrency issues we encountered, we decided to change this behavior.
Starting version 22.09.15 export format has been changed to a single JSON file.
It works by retrieving all Mongo export commands from all the plugins, extracting all the needed information, and running aggregation pipelines with the $merge stage to a single collection.
That way, on-demand app user data exports will create one collection with the exported data, and when the dashboard user downloads the export, it will just stream it from the collection to the output to the browser.
The end result would be the same documents, except in one single file and having one additional property named _col, which would indicate the original collection name from which this document came:
This way, we overcome all the problems we had with the previous export, and there are no changes for existing plugins to implement as data is extracted from the old export commands. Additionally, it should make the workflow with exports easier, as there is only one single file now to handle without the need to unarchive and process multiple files.
Hope you like the change and let us know your thoughts about it on our community discord server: https://discord.gg/countly