About

Test

Popular posts from this blog

Node.js: Optimisations on parsing large JSON file

Image
What if you have 14GB of JSON data, and you want to read it in Node.js and perform a filtering operation. How would you do it? Type of the data For the Backend DB architecture  project, with a friend we generated 15,930,001 posts for a total of 14GB of data. Each post has the following type: export class Post { public client : number ; //ID of the Client public channel : number ; //ID of the Channel public id : number public created_date : number ; public text : string ; public post_type : "TEXT" | "IMAGE" | "VIDEO" | "LINK" ; public labels : string []; //max 10 public insights : number []; //array of 100 integers } Baseline Good benchmark for the minimum time to scan the file is to compare with  the "wc" utility. It yields 12.1 seconds scan time for "\n": time wc -l ../posts.json 15930000 ../posts.json wc -l ../posts.json 8.23 s user 3.32 s system 94 % cpu 12 . 191 total Filtering for...
Read more

Backend DB architecture

Image
With a friend, we were trying to prepare for a Facebook general architecture interview question. I proposed we try to solve the following problem (very keen to my heart): Backend architecture for massive post analytics  So basically - selecting a DB that can efficiently query and scale well with the number of users. Data representation of a post Each post has the following meta-data: Customer ID Channel ID (i.e. the Facebook page ID, or Twitter account ID, etc) Created date - date & time when posted Post type - one of "TEXT", "IMAGE", "VIDEO", "LINK" Text Labels Has no labels Has ALL of the specified labels Has ANY of the specified labels The analytics-data is just: 100 integers Sample queries A list of possible queries All posts for a "Channel ID" for the last 30 days All posts for all "Channel IDs" in a "Customer ID" for a given "Post type" for the last 60 days All...
Read more

Automating page scraping with Selenium and Chrome

Image
Sometimes you are interested in information on some page, be it on some social media, or on a news site, or a blog. But you do not have an automated way via API to access this data. I.e. you want to stock the public FB posts on the girl that did not accept your friend request. Or you want to monitor a site for product availabily or price range. Or you might want to get notified if the university web site pushes a new update. Be warned that it might be agains the respective sites policy to make such automated scannings, or you might risk suspending your account or legal proceedings. But lately I have the need to monitor a web site for auto parts. I was interested in the availability of a certain part for my car. I wanted to get notified when there is something new. The vendor does not have API or RSS feeds, but claims that the site is updated the moment there is a new part available and the moment when an old part is sold. Fortunatelly the filter for figuring out the car, model, yea...
Read more