Upload file to Google Storage from memory with Node.js gcloud library

-
These days was interacting with the Google Cloud Storage(GCS) with the gcloud nodejs library(version 0.31.0) and was surprised that there is no API method to directly upload file from memory. So if you are looking for it - the shorthand is just to use the write stream returned from createWriteStream() on file object:

var gcloud = require("gcloud");
var storage = gcloud.storage({projectId:"dir-bg-scraper"});

var sample_TXT_file = "Hello there,\r\n"+
          "This is sample text file with important info\r\n\r\n"+
          "Best Regards!";

storage.createBucket("dir-bg-scraper", function(err, bucket, apiResponse) {
    if (err===null) {
        // bucket is the newly created "Test-Bucket"
        console.log("'dir-bg-scraper.appspot.com' successfully created!");

        // create 'README.md' file into this bucket
        var new_file = bucket.file("README.md");
        upload_file(new_file, sample_TXT_file, function(err){
            if (err!==null) {
                console.log("Successfully uploaded the file!");
            } else {
                console.error("Failed to uploaded the file..");
            }
        });
        
    } else {
        console.error("Feiled to create 'dir-bg-scraper.appspot.com'..", err);
    }
});

function upload_file(file, contents, callback) {
    // open write stream
    var stream = file.createWriteStream();

    // if there is an error signal back 
    stream.on('error', callback);

    // if everything is successfull signal back
    stream.on('finish', callback);

    // send the contents
    stream.end(contents);       
}

In this sample the "dir-bg-scraper" happens to be the name of my project. So change it with your name. I am accessing the GCS from my local laptop and am running the program in enviromenent already initialized and configured with the project. That is why there is no specific authentication.

Also keep in mind that buckets are shared across aaaaall projects and must be unique. This means that you might need to try multiple times until you come across empty bucket name.

The function that does the work is "upload_file".

Comments

Post a Comment

Popular posts from this blog

Data types: Backend DB architecture

-
Image
In this post, the data types that represent the entities mentioned in " Backend DB architecture " would be presented. The language is Typescript running in Node. Lets start with the Client: export class Client { public id : number ; public name : string ; labels : string []; constructor () { } } The `labels` array contains all labels that could happen on any of the `Post`s. A `Client` could have any number of `Channel`s associated with it. The `_client_ref` points to the `Client`:: import { Client } from "./client" ; export class Channel { public client : number ; public id : number ; public name : string ; public _client_ref : Client ; constructor ( client : Client ) { this . _client_ref = client ; } } Each `Channel` could have any number of `Post`s: import { Client } from "./client" ; import { Channel } from "./channel" ; export class Post { public client : number ; public chann...
Read more

Node.js: Optimisations on parsing large JSON file

-
Image
What if you have 14GB of JSON data, and you want to read it in Node.js and perform a filtering operation. How would you do it? Type of the data For the Backend DB architecture  project, with a friend we generated 15,930,001 posts for a total of 14GB of data. Each post has the following type: export class Post { public client : number ; //ID of the Client public channel : number ; //ID of the Channel public id : number public created_date : number ; public text : string ; public post_type : "TEXT" | "IMAGE" | "VIDEO" | "LINK" ; public labels : string []; //max 10 public insights : number []; //array of 100 integers } Baseline Good benchmark for the minimum time to scan the file is to compare with  the "wc" utility. It yields 12.1 seconds scan time for "\n": time wc -l ../posts.json 15930000 ../posts.json wc -l ../posts.json 8.23 s user 3.32 s system 94 % cpu 12 . 191 total Filtering for...
Read more

Back to teaching

-
When I was still a student at Sofia University, Bulgaria back in 1999 I got an assistant position within the Faculty of Mathematics and Informatics. It was so much involving and enjoyable for me that I went on and did it for 6 semesters up till I finished with my bachelor degree. This is not uncommon practice since the “students as teachers” tend to bring a lot of enthusiasm to the classroom. This was a very positive experience. It shaped and improved skills like talking in front of an audience, showing leadership, reaching out and inspiring the inner curiosity, feeling the audience with the most appropriate pace for delivering the content. Of course back then I did not do it with the aim of self improving. I did it for the fun and the prestige in the eyes of friends, colleagues and family that comes with it. I realize the above mentioned added values involved with such activity only later, when looking back and musing over it. And it was a lot of fun, especially after the kind of ...
Read more