5.7 Evaluate reactions on Linkedin
Measure the success of the LinkedIn reactions feature
As human emotions, aren't always binary; LinkedIn just like Facebook launched the reaction feature on its platform. I will take this question as a Product Management Interview question as I was asked recently in a PM interview, and I will evaluate the impact of this feature and the impact it puts on the entire LinkedIn ecosystem separately.
I would divide to divide of the goal of using this feature into two verticals.
1. Customer Goal 🍮
Providing customers with more options in addition to “Like” to express themselves – Human emotions have a wide range and variety. It isn’t binary. The Like button limits the range of emotions that someone can express while engaging with content on their news feed.
Increasing social validation. Giving users more ways to quickly and constructively communicate with one another – When users post on LinkedIn, they want more ways to feel heard and understand why someone has liked what they have said.
2. Business goal 🧑💼
Increase engagement of users on the news feed- By engagement, I mean – the average number of shares, posts, likes, reactions, comments per visitor during a 30-day period.
Retention – For the test group that is exposed to reactions. – Do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
Testing the reaction feature 🖥️
I would use the following three metrics to test the feature of reactions.
🏸 Discoverability/Acquisition: Are people able to discover “Reactions”? It is currently hidden under the Like button- is it easily discoverable?
🎢 Usage: For the visitors that discover Reactions – How many people are clicking on the reactions? What is the total number of clicks? I would measure this for at least a week. I would also measure the % of users who are using likes compared to reactions?
🤘🏼Usage by reaction type: How many clicks per reaction type? This would help me understand if I even have the right set of reactions in there or do, I need different reactions?
Testing the impact of reactions on the entire ecosystem 😻
🎮 Usage/Engagement/Content Creation – For the user group that is exposed to reactions (Test Group)– does their average activity during a 30 day period increase due to reactions – Are they liking/reacting/commenting/sharing more as compared to people (Control group) who do not see reactions? Are they creating more content than before the test (pre-post analysis) and as compared to the control group (A/B)?
🦋 Retention – For the test group that is exposed to reactions – do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
🐧 Engagement – Does it increase the time spent and length of the scroll on the news feed? Test A/B and see if there is a statistically significant difference?
💲Monetization – Is there a higher click-thru rate (CTR) on ads for visitors who were exposed to reactions?
Interviewer (I): You mentioned A/B testing many a time. How would you prevent the control group from seeing reactions in their feed if their friends are in the test group?
Response:
- I would ensure that the control and test groups are far apart in the LinkedIn social graph.
- If they are in the same social graph and reactions are encountered by the control group, I would ask the engineering team to convert the reactions to like for the test groups.
Interviewer (I): How would you differentiate that whether passive users have become active or active users or highly socially engaged people” also had a marked improvement in their social activity due to the launch of this feature.
Response: To measure incremental improvement in engagement, I would take people in test and control groups with similar levels of prior engagement and then compare both.