Posts

Showing posts from May, 2020

Node.js: Optimisations on parsing large JSON file

Image
What if you have 14GB of JSON data, and you want to read it in Node.js and perform a filtering operation. How would you do it? Type of the data For the Backend DB architecture  project, with a friend we generated 15,930,001 posts for a total of 14GB of data. Each post has the following type: export class Post { public client : number ; //ID of the Client public channel : number ; //ID of the Channel public id : number public created_date : number ; public text : string ; public post_type : "TEXT" | "IMAGE" | "VIDEO" | "LINK" ; public labels : string []; //max 10 public insights : number []; //array of 100 integers } Baseline Good benchmark for the minimum time to scan the file is to compare with  the "wc" utility. It yields 12.1 seconds scan time for "\n": time wc -l ../posts.json 15930000 ../posts.json wc -l ../posts.json 8.23 s user 3.32 s system 94 % cpu 12 . 191 total Filtering for...

Data types: Backend DB architecture

Image
In this post, the data types that represent the entities mentioned in " Backend DB architecture " would be presented. The language is Typescript running in Node. Lets start with the Client: export class Client { public id : number ; public name : string ; labels : string []; constructor () { } } The `labels` array contains all labels that could happen on any of the `Post`s. A `Client` could have any number of `Channel`s associated with it. The `_client_ref` points to the `Client`:: import { Client } from "./client" ; export class Channel { public client : number ; public id : number ; public name : string ; public _client_ref : Client ; constructor ( client : Client ) { this . _client_ref = client ; } } Each `Channel` could have any number of `Post`s: import { Client } from "./client" ; import { Channel } from "./channel" ; export class Post { public client : number ; public chann...

Backend DB architecture

Image
With a friend, we were trying to prepare for a Facebook general architecture interview question. I proposed we try to solve the following problem (very keen to my heart): Backend architecture for massive post analytics  So basically - selecting a DB that can efficiently query and scale well with the number of users. Data representation of a post Each post has the following meta-data: Customer ID Channel ID (i.e. the Facebook page ID, or Twitter account ID, etc) Created date - date & time when posted Post type - one of "TEXT", "IMAGE", "VIDEO", "LINK" Text Labels Has no labels Has ALL of the specified labels Has ANY of the specified labels The analytics-data is just: 100 integers Sample queries A list of possible queries All posts for a "Channel ID" for the last 30 days All posts for all "Channel IDs" in a "Customer ID" for a given "Post type" for the last 60 days All...

Any suggestions on how to “hack” headaches?

Image
This is short write up on a question in a Facebook group that I am member of. Q: Any suggestions on how to “hack” headaches (likely caused by muscle tension) ? Been suffering for almost 2 years, nothing helps. Have tried physiotherapy, osteopathy, zonetherapy, acupuncture, red light therapy, cryo, massages, floating, different medications. what would be a biohacker’s approach? Injurymap / CC BY (https://creativecommons.org/licenses/by/4.0) My general answer there, based on teachings of dutch  Wim Hof Iceman  and the russian professor  Buteyko My answer: I would suggest 20 minute holding your breath exercise. Why? To my understanding - headache means constricted blood vessels and reduced blood flow to the brain (i.e. low oxygenation). Low oxygenation / mitochondrial ATP production, to my understanding, is the base to all issues, not just headache. For example stress, with increased shallow breathing, would deplete CO2 in the blood. Which will constrict blood ...

Follow up on Tesla investment

Image
This is follow up on the " Tesla investment " post. Will do a retrospective on it. The blog was posted April, 23. And the shares were bought a day later. Before that there was good several weeks of trying to figure out how the brokerage system works. There for sure are many options both local and worldwide. Each comes with its advantages and risk. I am in no position to comment on this. I signed up with " Elana Trading " a local bulgarian broker (and it seems respected), for the simple reason that it has offices where I work. Not a great reason - but that is it. International stock come at 15$ charge (minimum) on opening and the same amount on closing. So having chosen a broker, I gathered my thoughts into the blog, and bought the shares. Each was $264 including the brokerage fee per share. What I did not realise is that this was only few days before Q1 2019 earnings results. Very nooby! As it turns out - notwinstanding much I love the company and the product...