Backend DB architecture

-
With a friend, we were trying to prepare for a Facebook general architecture interview question. I proposed we try to solve the following problem (very keen to my heart):
Backend architecture for massive post analytics 
So basically - selecting a DB that can efficiently query and scale well with the number of users.


Data representation of a post

Each post has the following meta-data:

  • Customer ID
  • Channel ID (i.e. the Facebook page ID, or Twitter account ID, etc)
  • Created date - date & time when posted
  • Post type - one of "TEXT", "IMAGE", "VIDEO", "LINK"
  • Text
  • Labels
    • Has no labels
    • Has ALL of the specified labels
    • Has ANY of the specified labels
The analytics-data is just:
  • 100 integers

Sample queries

A list of possible queries
  • All posts for a "Channel ID" for the last 30 days
  • All posts for all "Channel IDs" in a "Customer ID" for a given "Post type" for the last 60 days
  • All posts in a "Customer ID" that have specified "Labels" for the last year
  • All posts in a  "Channel ID" of  "Type"="TEXT" that has "no labels" in the last 6 months

POC parameters

We agreed that we should generate sample data for a Proof Of Concept. Each will choose a solution and will implement it. We would benchmark against each other. 

We should generate sample data set:
  • 5,000 Clients
  • 50,000 Channels - randomly distributed in the clients
  • 15 Gigabytes of total 16,040,001 posts. Each posts has:
    • maximum of 10 "Labels"
    • "Created date" - after 2018


Table of Contents for the series



Comments

Post a Comment

Popular posts from this blog

Data types: Backend DB architecture

-
Image
In this post, the data types that represent the entities mentioned in " Backend DB architecture " would be presented. The language is Typescript running in Node. Lets start with the Client: export class Client { public id : number ; public name : string ; labels : string []; constructor () { } } The `labels` array contains all labels that could happen on any of the `Post`s. A `Client` could have any number of `Channel`s associated with it. The `_client_ref` points to the `Client`:: import { Client } from "./client" ; export class Channel { public client : number ; public id : number ; public name : string ; public _client_ref : Client ; constructor ( client : Client ) { this . _client_ref = client ; } } Each `Channel` could have any number of `Post`s: import { Client } from "./client" ; import { Channel } from "./channel" ; export class Post { public client : number ; public chann...
Read more

Node.js: Optimisations on parsing large JSON file

-
Image
What if you have 14GB of JSON data, and you want to read it in Node.js and perform a filtering operation. How would you do it? Type of the data For the Backend DB architecture  project, with a friend we generated 15,930,001 posts for a total of 14GB of data. Each post has the following type: export class Post { public client : number ; //ID of the Client public channel : number ; //ID of the Channel public id : number public created_date : number ; public text : string ; public post_type : "TEXT" | "IMAGE" | "VIDEO" | "LINK" ; public labels : string []; //max 10 public insights : number []; //array of 100 integers } Baseline Good benchmark for the minimum time to scan the file is to compare with  the "wc" utility. It yields 12.1 seconds scan time for "\n": time wc -l ../posts.json 15930000 ../posts.json wc -l ../posts.json 8.23 s user 3.32 s system 94 % cpu 12 . 191 total Filtering for...
Read more

Back to teaching

-
When I was still a student at Sofia University, Bulgaria back in 1999 I got an assistant position within the Faculty of Mathematics and Informatics. It was so much involving and enjoyable for me that I went on and did it for 6 semesters up till I finished with my bachelor degree. This is not uncommon practice since the “students as teachers” tend to bring a lot of enthusiasm to the classroom. This was a very positive experience. It shaped and improved skills like talking in front of an audience, showing leadership, reaching out and inspiring the inner curiosity, feeling the audience with the most appropriate pace for delivering the content. Of course back then I did not do it with the aim of self improving. I did it for the fun and the prestige in the eyes of friends, colleagues and family that comes with it. I realize the above mentioned added values involved with such activity only later, when looking back and musing over it. And it was a lot of fun, especially after the kind of ...
Read more