TL;DR: How to use feature flags to deploy to production rapidly without breaking things
There is always a trade-off between trunk-based development (where developers frequently merge small changes to the main branch) and long-lived feature branches (where developers work on a branch and merge to the main branch only when work is ready to go live).
The advantages of trunk-based development are obvious: simple – you just branch off main, develop and then send pull requests to main; more agile – i.e. continuously delivery of smaller chunks of work to production.
The disadvantages are more subtle: trunk-based development works perfectly if only one person is working on a repo at a given time. When there are multiple developers or multiple teams trying to push features into production in parallel, chaos ensues. The moment you do a product release for your own feature, everything that other developers have merged to master since the last release gets automatically piggybacked to production. At best, users see features (or half-baked bits of features) they’re not supposed to see yet. At worst, something breaks in production due to a commit that was merged to main but not yet intended to go to production.
The tempting solution is long lived feature branches: each team branches off the main branch and continue development on a separate feature branch. They may deploy that branch to a test environment for testing, but never to staging or production. Only once testing is completed and the work is ready to go live, will the team now merge the long lived branch back to main. This ensures the rule “if it’s in main, it’s ready to go live”.
This theoretically elegant solution becomes a nightmare in practice. First, while you were working on your long lived, branch, other teams may have merged their own long lived branches to main, making the diff between your branch and main not only large, but rife with conflicts. Such a big change to the master branch will always require a full regression to ensure that nothing’s broken. Second, waiting for a full feature to be developed and tested before merging or deploying, means that deployments are large and infrequent. They’re the opposite of “continuous” delivery. They become “releases”, and the nature of large releases is that they become more like a rocket launch the bigger they are: massive amounts of testing, lots of questions from managers and business people, go/no-go meetings, post-release verifications and various other processes to mitigate the risk of doing big changes to production.
At :Different, we’ve tried both approaches and have experienced the downsides of both. But what we wanted was the upsides of both: the simple, rapid merging and deployment of trunk based development, with the safety of long lived branches. So we naturally opted for trunk based development with feature flags.
First of all, a feature flag in our world is extremely simple: a boolean variable stored in the environment’s database, loaded onto the application’s (front-end and backend both) memory in a way that any part of the code can read it safely. Then where new development takes place, the new code goes in the if block, the old code goes in the else block. Once the development is completed and the feature is generally available, we remove the database entry, all references to the feature flag variable in code, along with the old code. At the most fundamental level, that is all what a feature flag is. We don’t use products or libraries such as Optimizely.
Say you’re developing a blog application with a JavaScript single page application as the front-end, a Node.js API as the backend and MongoDB as the persistence layer. A new feature you’re working on is “Emoji reactions on blog posts”. You want your developers to be able to test and merge small bits of development to the main branch, but at the same time, not accidentally relase half-baked bits to production.
{
captcha: true, // a previous feature
emojis: false // always default to false, to avoid accidental enabling in prod
}
// register new route to handle emoji reactions
if (flags.emojis) {
app.post('/posts/:id/emojis', (req, res) => {
// code here
});
}
// update any existing code
app.get('/posts/:id', (req, res) => {
// existing code
if (flags.emojis) {
// new code to maybe pack emojis into the response
} else {
// some old code that is obsoleted by emojis, maybe
}
});
You need new database structures (tables, columns, collections, objects, or new fields inside objects). Persistent data cannot be feature flagged. So you need to update your database so that the schema works whether the flag is on or not. In this case, an example:
New reference data collection: emojis:
{
"thumbs_up": 👍,
"thumbs_down": 👎 ,
...
}
And all entries in the post collection should at least have an empty “emoji” object.
An old post object, after running the emoji data migration.
{
"id": 1234,
"title": "This is a post",
"body": "...",
"emojis": []
}
Assuming you already have a means of fetching environment’s feature flags:
// on page load
flags = api.fetch('/feature-flags');
Now in the UI:
const Post = (props) => {
const flags = useContext(FlagContext);
return (<div>
<h1>{props.title}</h1>
<div>{props.body}</div>
{ flags.emojis && pros.emojis.map((emoji) => (<span>{emoji}</span>)) }
</div>);
};
Make sure the feature flag is false in production, then start developing, merging and deploying to test environments.
The feature flag can be true in your test environments.
Give QA and product managers a simple interface to enable/disable feature flags in test environments so that they can run comparison tests without bothering you.
This is a more advanced thing. Let’s say you want to enable the feature for a subset of ‘guinea pig’ users. Maybe in the QA environment, that’s a small group of QA engineers. Maybe in production, that’s a small number of trusted beta customers. How will you modify your feature flag scheme?
Assume you have this in the backend:
app.get('/posts/:id', (req, res) => {
// existing code
if (flags.emojis && getCookie(req, 'ff_emojis')) { // only if emojis are enabled at env level AND user level
// new code to maybe pack emojis into the response
} else {
// some old code that is obsoleted by emojis, maybe
}
});
And you send it as a cookie:
Cookie: ff_emojis=1;
If necessary, this flag information can be stored in a more granual flag collection and served to the front end:
// user_id -> flags (user level flags)
{
johndoe: { captcha: true, emoji: false},
ksmith: { captcha: true, emoji: true}
}
Once your full feature is in production behind a feature flag, let your product manager run some tests (maybe by enabling the flag only for themselves). Once they’re satisfied, they’ll enable the flag in production globally.
Once the feature runs for a while and there are no bugs, you need to make sure that you remove all references to the feature flag from your system: front-end code, back-end code, database and configurations. Otherwise your code will get unreadable due to flag driven if-else statements.
{
//captcha: true,
//emojis: false
}
app.post('/posts/:id/emojis', (req, res) => {
// code here
});
app.get('/posts/:id', (req, res) => {
// existing code
// new code to maybe pack emojis into the response
});
const Post = (props) => {
const flags = useContext(FlagContext);
return (<div>
<h1>{props.title}</h1>
<div>{props.body}</div>
{ pros.emojis.map((emoji) => (<span>{emoji}</span>)) }
</div>);
};
Repeat for every new feature. Now you’re ready to do trunk based development with feature flags.