About

Test

Popular posts from this blog

Node.js: Optimisations on parsing large JSON file

Image
What if you have 14GB of JSON data, and you want to read it in Node.js and perform a filtering operation. How would you do it? Type of the data For the Backend DB architecture  project, with a friend we generated 15,930,001 posts for a total of 14GB of data. Each post has the following type: export class Post { public client : number ; //ID of the Client public channel : number ; //ID of the Channel public id : number public created_date : number ; public text : string ; public post_type : "TEXT" | "IMAGE" | "VIDEO" | "LINK" ; public labels : string []; //max 10 public insights : number []; //array of 100 integers } Baseline Good benchmark for the minimum time to scan the file is to compare with  the "wc" utility. It yields 12.1 seconds scan time for "\n": time wc -l ../posts.json 15930000 ../posts.json wc -l ../posts.json 8.23 s user 3.32 s system 94 % cpu 12 . 191 total Filtering for...
Read more

Data types: Backend DB architecture

Image
In this post, the data types that represent the entities mentioned in " Backend DB architecture " would be presented. The language is Typescript running in Node. Lets start with the Client: export class Client { public id : number ; public name : string ; labels : string []; constructor () { } } The `labels` array contains all labels that could happen on any of the `Post`s. A `Client` could have any number of `Channel`s associated with it. The `_client_ref` points to the `Client`:: import { Client } from "./client" ; export class Channel { public client : number ; public id : number ; public name : string ; public _client_ref : Client ; constructor ( client : Client ) { this . _client_ref = client ; } } Each `Channel` could have any number of `Post`s: import { Client } from "./client" ; import { Channel } from "./channel" ; export class Post { public client : number ; public chann...
Read more

Backend DB architecture

Image
With a friend, we were trying to prepare for a Facebook general architecture interview question. I proposed we try to solve the following problem (very keen to my heart): Backend architecture for massive post analytics  So basically - selecting a DB that can efficiently query and scale well with the number of users. Data representation of a post Each post has the following meta-data: Customer ID Channel ID (i.e. the Facebook page ID, or Twitter account ID, etc) Created date - date & time when posted Post type - one of "TEXT", "IMAGE", "VIDEO", "LINK" Text Labels Has no labels Has ALL of the specified labels Has ANY of the specified labels The analytics-data is just: 100 integers Sample queries A list of possible queries All posts for a "Channel ID" for the last 30 days All posts for all "Channel IDs" in a "Customer ID" for a given "Post type" for the last 60 days All...
Read more