Slackalytics Part Two: Looping Back on Threads

In last month's installment of Ask a Data Scientist we introduced the CRISP-DM process and walked through the initial step of identifying a business problem, which in our case was "why are employees leaving important channels in Slack?" 

We were able to pull the content of the messages in Slack and produce Word Clouds using R to gain an understanding of the data.  This month we worked to see if we could quantity the problem of people leaving channels. 

Collecting the Data

In the weeks following our last installment of Slackalytics, we circled back to other message SUBTYPEs in Slack that we had previously excluded:

slack_messages<-sqlQuery(ARKABASE,
                        "SELECT *
                         FROM ARKATPR.SLAK_TPR.SLACK_MESSAGES")
sqlQuery(ARKABASE,
          "SELECT DISTINCT SUBTYPE 
          FROM ARKATPR.SLAK_TPR.SLACK_MESSAGES")

##              SUBTYPE
## 1      channel_topic
## 2         file_share
## 3       channel_name
## 4        pinned_item
## 5        bot_message
## 6            bot_add
## 7               <NA>
## 8         bot_remove
## 9      channel_leave
## 10      reminder_add
## 11      channel_join
## 12   reply_broadcast
## 13                  
## 14      file_comment
## 15   channel_archive
## 16 slackbot_response
## 17   channel_purpose
## 18      file_mention

In order to determine why employees are leaving channels we needed to identify which channels were being left and by whom, so we focused on 2 specific SUBTYPEs:

##              SUBTYPE
## 9      channel_leave
## 11      channel_join

Our data scientists used Tableau to analyze this time-series data and were able identify the top channels joined, top channels left, top joiners and top leavers.

The Business Question

After presenting the data to the team, we began to ask questions about Slack and it's effectiveness. One of the common questions asked, "Is Slack more of a distraction than a value-added tool?" 

In order to answer that question, we need to answer the following questions:

• How many people in channels are contributing to the conversation and how many are observing?
  • If there are people who are not contributing to certain channels, why are they there? Are they being distracted by the conversations in the channel? Are the observers spending time reading the conversations?
• Are Slack messages replacing email?
  •  If Slack is replacing email and questions are answered quicker then that adds value.
• What does the behavior of joiners/leavers look like? 
  • Why are Chris and Jared leaving and joining so many channels? (refer to data viz above)  

Lack of Data

We turned to the data to see if we could answer any of our business questions.

How many people in channels are contributing to the conversation and how many are observing? 

Our data scientists reviewed the Slack API to determine if we could determine who contributes and who observers in each channel. We are already able to accurately determine who contributes, but found that it is not feasible to determine if observers are reading conversations or not without creating an on-going stream of Slack status by channel by user, on-going. This would be a considerable additional investment without a certain payback, so we determine this was not an area of further research.

Are Slack messages replacing email?

In order to measure this we would need to measure whether email volume has decreased since the implementation of Slack. This is extremely difficult to measure for two reasons. The first reason is that bringing our email traffic into our data warehouse would require replicating the security groups we have implemented on out email platform in our warehouse. The second reason is that over half of the company was employed at Arkatechture after Slack was introduced, meaning we can't compare their use of email before Slack. Therefore, this also was not an area of further effort.

What does the behavior of joiners/leavers look like?

We noticed that Chris and Jared are both top joiner & leavers, so to understand this behavior better, we decided to interview them in order to find out what motivates them to leave or join channels and see if we can validate this in the messages.

Changing the Business Question

After we started the Slackalytics project, Slack introduced "threads" to their platform. Threads allow users to reply directly to messages in any Slack channel. Unless tagged in the thread, members of a channel will not be notified when someone responds in a thread. Between the lack of data and the introduction of threads, the consensus user experience was that this enhancement made Slack significantly more efficient, since it added further structuring to the content of a channel.  As a result, we decided to re-examine our business question "Is Slack more of a distraction than a value-added tool?".

To re-examine this, we needed to update our integration to the Slack API to add an argument to identify the threading and will then review both the joiner/leaver trends and volume of threaded vs. non-threaded messages to determine if there is a measured difference in user behavior after the introduction of threads compared with before.

Check back to see how our user behavior compares before & after the introduction of threads and if further analytics on this business problem are justified!