Creating a community and social media dashboard using Dashing and Keen.io

Last year I showed some people on Twitter and over at EMC World how you can create a cool dashboard with Dashing that shows a ton of stuff from an infrastructure perspective, and it was well received. A little over a month ago I started looking at the dashboard again but from another perspective, looking for social media interactions and community metrics. Last but not least we wanted to share this with the community, and I’ll explain how we created what is now live up and public here and what later became an interesting tool for analytics, using Keen.io as a backend.

dashboard_example

First off, Dashing is a beautiful dashboard created by the nice people at Shopify. It’s available for free and it’s very easy to get started with, just read the docs. When you have it up and running you can then feed data into the dashboard and it will auto-update in real-time. What’s really cool about this dashboard is that you can get data both from local jobs running on the server where you’re running the dashboard, and from external sources such as scripts or logging tools. I wanted our dashboard to be as simple as possible so I created a few jobs that run locally now with the possibility of adding external sources as well.

For the metric collection Dashing actually already has a lot of third-party created widgets that you can take and modify as you need. I’ve used a few like the Twitter and YouTube ones and also created a few that I needed for Docker Hub, StackOverflow and Constant Contact.

With that we were able to get a good current overview of how our community metrics were stacking up and see if we had a good response to different methods of reaching out to the community, and it was a great start. But we also wanted to add some way of storing the data and do some basic analytics of it. I looked into using Redis as a backend but really didn’t want to handle the storage part and neither did I want to build my own analytics tool. So I looked around and the always smart Matt Cowger told me to look at keen.io, so I did.

dashboard_analytics

Keen.io is a really cool app where you can store data using simple REST queries and then retrieve the data with analytics applied, super smart! So I looked into how and if I could apply and use this with Dashing, but didn’t find anyone that had done it before. So I dug a little deeper and found that keen.io has a Ruby gem, which I could then just add into the Dashing jobs and start feeding the data collected for the dashboard into keen.io.

Let’s look at an example of a Dashing job that’s been enabled for keen.io, I’m using the Twitter one from foobugs as an example, edits are in bold.

#!/usr/bin/env ruby
require 'nokogiri'
require 'open-uri'
require 'keen'

# Track public available information of a twitter user like follower, follower
# and tweet count by scraping the user profile page.

# Config
# ------
twitter_username = ENV['TWITTER_USERNAME'] || 'emccode'

SCHEDULER.every '60m', :first_in => 0 do |job|
 doc = Nokogiri::HTML(open("https://twitter.com/#{twitter_username}"))
 tweets = doc.css('a[data-nav=tweets]').first.attributes['title'].value.split(' ').first
 followers = doc.css('a[data-nav=followers]').first.attributes['title'].value.split(' ').first
 following = doc.css('a[data-nav=following]').first.attributes['title'].value.split(' ').first

 send_event('twitter_user_tweets', current: tweets)
 send_event('twitter_user_followers', current: followers)
 send_event('twitter_user_following', current: following)
 Keen.publish(:twitter, { :handle => twitter_username, :tweets => tweets.to_i, :followers => followers.to_i })
end

Yes, it’s that simple! But, you might say, how does it know who to store the information for? It’s not configured for any project, doesn’t have any access keys or anything! I’ll get to that.

First, add the following to your config.ru to make use of the dotenv gem. Make sure you put this at the very top!

require 'dotenv'
Dotenv.load
require 'dashing'
<snip>

Second, add the following to your Gemfile:

gem 'keen'
gem 'dotenv'

Then run “bundle” to install the dotenv and keen gems.

Now all you have to do is to store your keen.io variables in a file called .env at the root of the dashing directory and you’re good to go, like this:

KEEN_PROJECT_ID=yourprojectid
KEEN_MASTER_KEY=yourmasterkey
KEEN_WRITE_KEY=yourwritekey
KEEN_READ_KEY=yourreadkey

Now start the dashboard and you should see data coming in both to your awesome dashboard and to your project over at keen.io. The metrics can then be visualized by using their excellent examples over here, and you can see what we’re currently using for EMC {code} over here. Pretty cool I must say:)

Enjoy!

About Jonas Rosland

Developer Advocate at EMC {code}
This entry was posted in Analytics, EMC, Social Media, Visualization. Bookmark the permalink.

3 Responses to Creating a community and social media dashboard using Dashing and Keen.io

  1. Pingback: Smooth dashboards and easy analytics | EMC {code} – Blog

  2. Incredibly cool work Jonas!

    Question: Can you add a link to the Keen.io dashboard directly from pretty page on Dashing? I’m curious whether you intentionally leave that for behind the scenes or whether it’s something you’d like to show off as well. I’m inclined to let people scroll around the data bits:)

    Great stuff! My turn to dive in next!

    • Jonas Rosland says:

      Hi Matt!

      Thank you:)

      Yes, you can add links to the dashboard as you want, it’s standard HTML so you can change it in any way you want.
      It was intentionally left behind for now, you are correct:)
      The link to the dashboard is in the blog text but for those who just want it now you can find it here: http://dashboard.emccode.com/analytics.

      Have fun and explore!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s