Social Media Analytics on a Budget

Analyze Twitter in Minutes

Posted 3 months ago by Tom Maiaroto.


I've built quite a few social media analytics tools over the years (Social Harvest, Virality Score, etc.) ranging from simple to complex to just what the client needed to support their marketing campaign. Suffice to say I've learned the tools of the trade and have built systems that do everything the more expensive ones do such as; Sysomos, Radian6 (whatever they call themselves these days), Viral Heat, and more (sorry if you didn't get a shout out). There's plenty...And they're all expensive (well Viral Heat isn't really that pricey, but it's also fairly limited in comparison to the other giants). 

Literally, these companies are raking it for such simply data that's actually free! It's freely available. The cost is in the processing power to harvest (not as bad as you might think if you're clever), storing that data, and then taking the time to provide a nice user experience with useful visualizations. Is that worth thousands of dollars per month? No. It's not...Because there's no secret sauce there (that's why I focused on Virality Score, which is far more complex and took far longer to develop and test...It's an actual algorithm). 

All is not lost and you're in luck! You too have useful social media analytics within your reach even if you can't afford these expensive services.

Yes, you can't get around the storage and hosting costs. However, it doesn't have to be real expensive. You could, for example, use Digital Ocean for $10 or $20/mo. (Elastic Search is memory intense and you might have trouble for a $5/mo. server with only 512MB of RAM) and have a powerful enough server with enough storage. Then all you need is software...and guess what? It's all free, open source, software.

I mentioned Elastic Search. Yes, the search engine. As it turns out, it can be used for social media analytics as well. There exists a "river" (word for plugin that feeds data to the search engine to index) for Twitter! Typically they're made for various databases, but this one feeds from Twitter. Awesome!

You can spin up a server and issue a few commands to install Elastic Search and that river plugin. The installation and then configuration to actually index the data (you'll need a Twitter app setup, etc.) isn't the most user friendly...But it's also not that bad. I may even create a guide/tutorial for it in the future. Just know that if you know what you're doing you can set it up fairly quick. You get to specify the accounts to follow, keywords, etc. Best of all, you own all the data. It won't be locked away somewhere on some service that you need to pay.

What happens after you start indexing all this data from Twitter? You visualize it of course! How? Easy. The Elastic Search folks also put out, Kibana, a customizable dashboard application (can also be installed as a plugin). Look, their site even says no code required! It's true, it's true.

So once you have that, the configuration of your dashboard will take some time, but again nothing crazy once you get used to the tool. I will say that all of this is for the adventurous, but once setup you're pretty much golden. Just kick back and wait a few hours and data should be rolling in (assuming people are active in discussing whatever it is you are monitoring).

What does the end result look like? Here's a screen shot for an example.

From this quick dashboard you can already get a sense for the kind of data available to you from just Twitter. Popular hashtags, languages, locations, and overall mentions. You can narrow down everything by time or other criteria. Kibana makes it trivial to answer the question, "What was going on during this big spike here?" You can compare various Twitter users against each other or make any kind of search you want. Remember it's a search engine!

If you were feeling even more adventurous, you could add new panels to this dashboard. Kibana is open source and is built using the wonderful AngularJS framework. Though it has a lot of great features out of the box and will allow you to configure and save multiple dashboards to use and share with others. You can even lock them down from being edited farther.

But what about the full firehose?

I've said this many times in the past. The firehose isn't the most important thing. In fact, it actually very limited application if you're familiar with statistics. First, it can't search backwards. Second, and most important, I'll give you an analogy. If you wanted to determine the best mountain to mine, would you dig up each of those mountains in their entirety to figure out how much metal, stone, and other resources were there? No, of course not. You'd take a core sample and you'd be able to approximate. The exact same thing is true for social media.

It's actually quite rare that you'd need everything and the cost to store everything and the price you'd have to pay for the firehose...It's not worth it. Trust me. A million times over it is not worth it (99% of the time). 

Awesome, so what's left?

Well, remember this isn't a complete solution. It is only one social network (no, there appears to be no other rivers for other social networks at this time) and there's no advanced permissions on Kibana. You'd have to build in a login system with various access control features if you wanted and that takes time.

You don't have sentiment analyzed. Or gender. Geo-location is limited, but somewhat present as you can see here. So you could work on the data some more to be honest. However, this is awesome and it is going to give you a huge chunk of features (for free - aside from hosting costs) that you would otherwise be spending hundreds or even thousands of dollars per month on.

Other than those caveats, if you were serious about keeping this data around for the long term...You'd need some backup solutions and would eventually need to move data to Amazon S3 or something to recover disk space. You may need to eventually upgrade the size of your server as well. You might want to hook this up to your CMS or some other web app to add more features like e-mail alerts, etc.

I've listed quite a few things that are lacking, but nothing is perfect and the overwhelming value of this trick for the amount of time you'll invest simply outweighs all of that.

You are literally creating a social media analytics tool in minutes...A tool that took months for a team of developers to build and test. Why? Because you are leveraging those months of work from open source. So thank the kind Elastic Search folks.

The information can be powerful useful if used properly and you can really put together story with it as you can clearly visualize the big picture. Remember, big data doesn't have to be big, it just has to be effective.


Filed under
comments powered by Disqus

Search for Posts

Popular Labels

social media virality score general internet web development reviews and opinions php web tools hosting lithium web design

Recent Posts