Michał’s notes

f insights

f insights is a statistics web app for Flickr. I started developing it back in 2015. My goal was to get a taste of developing a full scale project in Scala and get familiar with its ecosystem, but to do that while working on something interesting for me.

What it does

The app fetches from Flickr API the data of user’s uploads, favs and contacts and builds a dashboard consisting of four parts (views). These are:

  • I like – showing user’s favourite tags and photographers
  • My photos – showing user’s popular tags and most devoted followers
  • Timeline – showing activity summary
  • Analytics – plots section

Below I present screenshots from the application.

Architecture

The application consist of several logical components. A dashboard builder, that pulls the data from Flickr API, transform it and stores in Cassandra, is modelled using Akka Actors. Dashboard data is exposed with the REST API, built using Play Framework. Finally, the API is consumed by a SPA based on Backbone.

Back-end

Back-end is built on top of Play Framework. It has a lot of positive opinions, what was one of factors, that convinced me for trying it out. Play Framework is shipped with bunch of components, that would satisfy majority of needs of an average web project. Template engine, assets management, JSON library, WebServices client, oauth2 support, integration with DI libraries (e.g. Guice), different testing tools (I decided for Specs2), Akka. The framework is evolving persistently, recently huge effort was put into better modularization, e.g. some of aforementioned components became stand-alone projects.

I’ve divided code into the following layers: controllers (broken down further into REST endpoints and actors system), domain (entities + services) and infrastructure (DBAL, API client). This division is reflected in packages structure.

The domain isn’t too complex, but hiding Flickr API behind repository interface is tricky, and I gave up after a while. In the end, it can’t be replaced with other provider, and testing this part of the system wasn’t my priority. Other problem is passing through layers context and tokens, that are tightly coupled with the concept of API.

On later phase I decided for introducing Scalaz. The primary reason for that were Monad Transformers. Operating on Future[M[T]] is a bit cumbersome, and OptionT or EitherT can ease that pain a bit. Although I did a mistake, because I wanted to use these data types only locally and don’t expose them in the API, what caused a lot of back and forth transforming.

The dashboard builder is modelled using Akka Actors, as this is a process consisting of several stages, that can be partially parallelized, but while respecting Flickr API requests qouta. If I would be developing this part of systems again, I would probably decide for Akka Streams instead, as they are closer to the nature of the problem.

Persistence layer

I have decided for Cassandra only because I wanted to exercise schema modelling for NoSQL database. Otherwise it’s not a reasonable choice for an app having one user;)

I have chosen Phantom as a database driver, after looking at few possible options. It seemed to have a nice API following the Scala way. Phantom authors are active on Stackoverflow, so if something isn’t clear, you can find help.

Front-end

The UI has a form of SPA. While working on it, I was picking tools that I either already knew or that seemed to be easily adoptable, thus the tech stack may seem slightly dated. As the main framework, I chose Backbone + LayoutManager, Handlebars as the template engine. I included as well Bootstrap and jQuery. For drawing plots, I picked Google Charts. As the CSS preprocessor, Less. For managing packages, I use npm and Bower.

Even if I was not planning putting too much effort in the UI, I spent a fair share of time working on it. Currently front-end makes 30% of the codebase.

Ops

I didn’t decide for putting the application online, so till now I have been running application only locally. I stuck to the sbt configuration shipped with Play Framework. After some struggle with running Cassandra directly on my OS, I decided for containerizing it with Docker. After every master branch update the application is built on circleci (rather straightforward operation).

What’s next

I would like to introduce db migrations, what should make the setup process easier and enable containerization of the web app itself.

As of new features, possibly suggesting users to follow may be a next thing I will develop.