How We Rebuilt Countly for Real-Time Analytics at 100+ Billion Data Points

Countly’s original architecture was built around a simple idea: process an event, update the relevant metrics, and make the result available immediately.

MongoDB handled the detailed events, analytical aggregates, and operational data behind the platform. That model worked well for years. Then our customers began operating at a different scale.

They were collecting billions of events, retaining longer histories, and adding more custom properties. Their teams were also asking more complex questions of the same data.

At the same time, two expectations did not change: SDK requests had to remain fast, and new data had to appear in analytics quickly.

The architecture was being asked to ingest more, calculate more, and query more—through the same path. Adding capacity could delay the problem. It could not remove it. We needed to rebuild how an event moved through Countly.

The result is Countly 26.01: an architecture designed to scale beyond 100 billion data points and deliver performance improvements of up to 100×, while keeping customer data inside infrastructure they control. Here is the engineering story behind that change.

The problem: every workload competed in one system

In the previous architecture, an incoming event triggered several kinds of work.

Countly stored the detailed event. It updated the aggregates behind dashboards. It maintained user and application data. The same database also served analytical queries and background jobs. Each workload was manageable on its own.

The difficulty appeared when they grew together.

• SDK ingestion produced continuous, bursty writes.
• Aggregation produced many small updates.
• Interactive queries scanned larger event histories.
• Background jobs consumed resources for longer periods.
• Operational requests still needed predictable response times.

A traffic spike could affect aggregation. A large query could compete with ingestion. A scheduled job could take capacity away from an interactive API.

Scaling MongoDB gave the shared system more resources, but the workloads still competed inside it. It also forced customers to scale more than the current bottleneck. If ingestion needed additional capacity, they often had to expand the database resources used by querying and operational data too.

The cost of keeping analytics fast increased with the size of the whole system.

We did not need one larger bottleneck. We needed separate paths for separate workloads.

The fix: turn one processing path into a pipeline

We changed the architecture around one principle:

Accept each event once, then let specialized services process it independently.

The new path starts like this:

SDK → Ingestor → Kafka

Kafka then feeds two analytical paths:

1. Kafka Connect → ClickHouse for detailed, high-scale event analysis.
2. Aggregator → MongoDB for common, precomputed product metrics

The query layer can use the appropriate store for each question. This removed the assumption that ingestion, aggregation, and querying had to progress at the same speed or use the same database model. It also let us solve each scaling limit directly.

We protected SDK ingestion first

The first limit was at the front of the system. An SDK request should not slow down because a dashboard query or aggregation task is busy. In the old path, downstream pressure could move back toward ingestion.

Kafka gave us a clean handoff. The ingestor now focuses on receiving SDK traffic, validating the payload, normalizing the event, and publishing it to Kafka.

Downstream services consume the event separately. This changes what happens during a traffic spike.

Before, every part of the analytical path had to absorb the peak at the same time. Now Kafka buffers the difference between the rate of arrival and the rate of processing.

If one consumer falls behind, the result is visible consumer lag. The SDK-facing path does not have to wait for that consumer to finish its work. Operators can then scale the affected consumer instead of scaling the entire platform.

Kafka does not make analytical queries faster. It makes high-volume ingestion more predictable and prevents slower analytical work from controlling the event intake path.

That separation is what allows Countly to keep data moving during sustained or bursty traffic.

We stopped sending analytical workloads to a transactional design

The next limit was query performance.

MongoDB is effective for flexible application data and targeted document access. Large analytical queries behave differently.

Consider a query that asks:

Show purchases from the last 90 days by country, app version, campaign, and customer segment.

The answer may require scanning millions or billions of events. Yet the query needs only a small subset of the fields stored on each event. A document database still works with event documents containing many properties the query does not need.

ClickHouse stores data by column. It can focus on the columns used for filtering and aggregation. That reduces the amount of data read and improves compression and aggregation efficiency.

We therefore moved granular event analytics to ClickHouse. This was not simply a database replacement. We also changed how Countly organizes analytical data.

Frequently queried dimensions—such as application, event, user, and time—are stored in typed columns. Events are ordered by application, event, and timestamp, with time-based partitions for historical data.

These choices help ClickHouse skip data that cannot match a query. When a customer asks for one event in one application over a specific period, the engine does not need to inspect the entire history.

We preserved flexible events without creating uncontrolled schemas

Behavioral data is not uniform.

Customers add event properties, user attributes, campaign data, and custom segments. A rigid schema would limit the questions they could ask.

Creating a dedicated column for every possible property would also create uncontrolled schema growth. We needed analytical structure without losing event flexibility.

The new ClickHouse model combines both. Common dimensions remain typed and optimized. Flexible properties use ClickHouse’s native JSON representation.

Dynamic-path limits and bucketed shared storage keep large sets of custom properties manageable. The full event context remains available without turning every property into a physical column.

The result is a model that is structured where performance depends on structure and flexible where product analytics depends on flexibility.

ClickHouse can then grow from a single node to replicated and sharded deployments as volume and availability requirements increase.

We added batching without giving up real-time analytics

Moving events to ClickHouse created another engineering decision.

We could write every event directly to ClickHouse as soon as it arrived. That would minimize delay, but it would generate a constant stream of small inserts.

Column-oriented databases handle larger blocks more efficiently. Tiny inserts create more write overhead and background merge work.

We used Kafka Connect to bridge the two models. Kafka Connect reads events from Kafka, groups them into batches, and writes those batches into ClickHouse.

This creates a tunable balance between freshness and efficiency:

• Smaller batches make events queryable sooner but increase write overhead.
• Larger batches improve throughput but keep events in the stream longer.

Countly provides low-latency, balanced, and throughput-focused profiles.

A customer running real-time personalization can prioritize freshness. A large reporting deployment can use larger batches to reduce infrastructure cost.

Real time is no longer tied to one fixed processing model. Customers can choose the operating point that fits their workload.

We kept MongoDB instead of replacing everything

Once ClickHouse handled detailed analytics, we could have moved every workload to it. That would have recreated the original problem with a different database.

MongoDB remains well suited to Countly’s operational data, application configuration, platform state, and established analytical aggregates.

It also remains valuable for questions whose answers can be prepared in advance.

Dashboards repeatedly request metrics such as daily totals, monthly users, session data, and common dimensions.

Recalculating those answers from raw events every time would waste resources, even with ClickHouse.

The separate aggregator therefore consumes Kafka events and continuously updates prepared MongoDB views.

That gives Countly two optimized query paths:

• ClickHouse answers flexible questions over detailed events.
• MongoDB keeps common answers and operational data ready.

Countly’s query layer supports both adapters. A query can move to ClickHouse when columnar analysis provides an advantage, while existing MongoDB-backed behavior remains available. This avoided a risky all-at-once migration. It also kept the change behind a stable product experience. Customers ask a question. Countly selects the suitable engine.

We separated services so customers can scale the bottleneck

Separating the databases was not enough. The application processes also needed clear boundaries.

Countly 26.01 separates the major responsibilities:

• Ingestor: receives SDK events.
• Aggregator: maintains prepared analytical views.
• API: serves product and analytical requests.
• Job server: controls scheduled and background work.
• Frontend: serves the dashboard independently.

This gives operators a targeted response to load. More SDK traffic? Scale the ingestor. Growing consumer lag? Scale the relevant consumer. More analytical demand? Add API or ClickHouse query capacity. Heavy scheduled work? Adjust job concurrency without taking resources from live ingestion.

Aggregation can be tuned independently too. More frequent processing improves freshness. Larger batches improve efficiency. Customers can change that balance without scaling every other service.

This creates better failure isolation and a more predictable cost model. Resources go to the component under pressure rather than the entire deployment.

What happens to an event now

Consider a purchase event sent from a mobile application.

1. The ingestor validates it and publishes it to Kafka.
2. Kafka buffers and distributes it to independent consumers.
3. Kafka Connect includes it in a batch written to ClickHouse.
4. The aggregator uses it to update prepared MongoDB metrics.
5. The query layer chooses MongoDB for a common summary or ClickHouse for detailed analysis.
6. The job server handles scheduled work outside the live ingestion path.

The event enters Countly once. It then powers multiple analytical experiences through specialized paths. No single service has to ingest, aggregate, store, and query the event alone.

What Countly can do now

The outcome comes from these changes working together.

Query large datasets faster

ClickHouse reads the analytical columns a query needs. Ordering and partitioning reduce the data examined. MongoDB aggregates avoid raw scans for common dashboards. Together, these changes support performance improvements of up to 100×.

Scale beyond 100 billion data points

Kafka partitions and buffers the event stream. Kafka Connect batches high-volume writes. ClickHouse supports replicated and sharded deployments. Countly’s services scale independently. The platform no longer depends on one process or storage pattern growing in every direction.

Keep data fresh during traffic spikes

Kafka absorbs short-term differences between ingestion and processing. ClickHouse ingestion and MongoDB aggregation progress independently. Pressure in one path does not have to stop the other.

Control infrastructure cost

Columnar queries reduce unnecessary reads. Batching improves write efficiency. Pre-aggregation avoids repeated computation. Independent services let customers add capacity to the current bottleneck instead of scaling the entire platform.

Keep behavioral data under customer control

Kafka, ClickHouse, MongoDB, and Countly’s application services can all run inside the customer’s environment. Customers gain a modern streaming and analytical architecture without moving behavioral data into a vendor-controlled analytics cloud.

The result: scale without narrowing the questions

Countly 26.01 is not faster because we added one faster database.

It is faster because we removed the points where different workloads constrained one another.

Kafka protects ingestion. Kafka Connect makes analytical writes efficient. ClickHouse accelerates detailed analysis. MongoDB keeps common answers ready. Separate services make scale targeted and controllable.

Each change solves a limitation we encountered as customer datasets grew.

Together, they allow Countly to accept more events, keep analytics fresh, and answer more ambitious questions—without giving up self-hosted data ownership.

That is what makes the next 100 billion data points possible.