Experiments

Test it on some.
Ship the winner to everyone

An experiment serves different values of your app’s variables to a share of your users, measures which one converts better on goals you define from your own data, and turns the winner into the value everyone gets. No release, and no separate testing tool.

In this demo car app, clustered charging pins beat the classic map. The winner is one click from being what every driver sees.

Part of engagement

A test you can
take to the decision

Every experiment compares each variant with a control on the goals you set, and says how sure it is.

  1. How many ideas at once?

    8

    groups per experiment, the control included

    Test up to seven variants against what you serve today. Each one sets the same variables, up to eight, to its own values.

  2. What counts as better?

    3

    goals, one of them primary

    A goal is a conversion you define from profile properties, cohort membership, or the events people trigger during the test. Each is compared variant by variant.

  3. Is it really better?

    95%

    the bar for calling a winner

    A variant wins when it is at least 95% likely to beat the baseline and its improvement range clears your target. Otherwise the page says no winner, plainly.

Set up in minutes

Four steps
to a running test

A short wizard, saved as a draft until you start it.

  1. 01

    Basics

    Name it and choose when it stops: by hand, after a number of days, or once a variant reaches your target improvement.

  2. 02

    Targeting

    Enroll a share of your users, everyone or only those matching profile properties, cohorts or profile groups.

  3. 03

    Goals

    Up to three conversions to measure, from who people are and what they do during the test.

  4. 04

    Variants

    The variables to test and a value for each variant, with the control set to what you serve today.

From result to release

The winner becomes the value.
No ticket, no deploy.

Roll out the winner, or any variant you choose, straight from the results. The values your app fetches change for the people you decide.

As a conditional valueAs the default
Who gets itThe people the experiment targeted, through a new conditionEveryone, as the variable’s new default
When to use itThe test was for one segment, and should stay that wayThe test was for everyone, and the winner should be too
What it keepsThe default and every other conditionThe conditions the variable already had
Offered whenThe experiment had targetingAlways

Works with

Experiments, with the
rest of Countly

Every feature reads the same live data, so an answer in Experiments doesn’t stop there. These are the handoffs built in.

All features

Data ownership

Tested on your users,
measured on your data

Enrollment, values and conversions all run on the server that holds your analytics, so there are no experiment events to send anywhere else.

Data privacy

One less vendor

Experiments use the same SDK, profiles and cohorts as your analytics and your variables.

Self-hosted, or dedicated

Run Countly in your own cloud, your data center or an air-gapped network. Or let us run a single-tenant instance in the Google Cloud region you choose.

No default vendor access

On Enterprise, Countly has no access unless you grant it. On Flex, access is limited, granted by you and logged.

ISO 27001Certified
ISO 27017Certified
ISO 27018Certified
SOC 2Type II examined
TISAX®Participant*

*Countly Ltd. is a TISAX® Participant. TISAX® is a registered trademark of and governed by ENX Association. Learn more at enx.com/tisax

Try it

Every feature.
One data layer.