DEV Community: Manuel Romero

Cool Colab Notebooks 📒 to play with funny Deep Learning 🧠🤖projects

Manuel Romero — Sat, 18 Jan 2020 12:31:47 +0000

Hi, devs!
I have been studying and working on Machine Learning for last 3 years.
I usually check out last ML/Deep Learning papers and try to develop Google Colab Notebooks to reproduce and play with their projects because many of them are awesome and fun!

Below, I will do a list o some of last Notebooks I created. All of them allow you to interact with the pretrained models they offer:

1) 3D Ken Burns

TL;DR Use my Notebook to upload a picture an create a 3D effect over it!

This is a reference implementation of 3D Ken Burns Effect from a Single Image using PyTorch. Given a single input image, it animates this still image with a virtual camera scan and zoom subject to motion parallax. Should you be making use of our work, please cite our paper.

Colab Notebook:
Repo

2) Learning to Paint

TL;DR Use my Notebook to upload a picture and create a video with the painting process and the final result

Arxiv | YouTube
We show how to teach machines to paint like human painters, who can use a small number of strokes to create fantastic paintings. By employing a neural renderer in model-based Deep Reinforcement Learning (DRL), our agents learn to determinethe position and color of each stroke and make long-term plans to decompose texture-rich images into strokes.

Colab Notebook:
Repo

3) TwinGAN

TL;DR Use my Notebook to upload a selfie and get a picture with your Anime and/or cat avatar.

Unsupervised Image Translation for Human Portraits.
We provide two pre-trained models: human to anime and human to cats.

Colab Notebook:
Repo

4) GPT-2 from Colab with Javascript interface

TL;DR Use the famous GPT-2 (even the most powerful model of 1.5B params) to generate or complete your text.

Colab Notebook:
Repo

From callbacks to fs/promises to handle the file system in Node.js

Manuel Romero — Fri, 25 May 2018 06:34:49 +0000

In this post, I will show you the evolution of Node.js to work with the file system.
Let's start by creating a file:

const fs = require("fs");

fs.writeFile("/tmp/test.js", "console.log('Hello world');", error => {
    if (error) console.error(error);
    else console.log("file created successfully!");
});

If we wanted to avoid callbacks, before Node.js v8 we had to manually promisify the fs.writeFile function or using third party modules such as bluebird or Q.

Let's promisify manually and wrap in a function the above code:

const fs = require("fs");

const writeFilePromise = (file, data) => {
    return new Promise((resolve, reject) => {
        fs.writeFile(file, data, error => {
            if (error) reject(error);
            resolve("file created successfully with handcrafted Promise!");
        });
    });
};

writeFilePromise(
        "/tmp/test2.js",
        "console.log('Hello world with handcrafted promise!');"
    )
    .then(result => console.log(result))

    .catch(error => console.log(error));

With the arrival of Node.js V8, 'util.promisify()' alllowed us to convert I/O functions that return callbacks into I/O functions that return promises.

const fs = require("fs");
const util = require("util");

const writeFile = util.promisify(fs.writeFile);

writeFile("/tmp/test3.js", "console.log('Hello world with promisify!');")
  .then(() => console.log("file created successfully with promisify!"))

  .catch(error => console.log(error));

Let's see the async/await version:

const fs = require("fs");
const { promisify } = require("util");

const writeFile = promisify(fs.writeFile);

async function main() {
    await writeFile("/tmp/test4.js",
        "console.log('Hello world with promisify and async/await!');");

    console.info("file created successfully with promisify and async/await!");
}

main().catch(error => console.error(error));

In its latest release (Node.js V10), functions of the fs can return promises directly, eliminating the extra step and overhead of the old way. This due to its fs/promises API.

const fsp = require("fs/promises");

try {
    await fsp.writeFile("/tmp/test5.js", "console.log('Hello world with Node.js v10 fs/promises!'");
    console.info("File created successfully with Node.js v10 fs/promises!");
} catch (error){
    console.error(error);
}

Note I am using top level await (my await code is not inside and async function). This feature is experimental yet, so if you want to check it out, use the following flag: -- experimental-repl-await.

The fs/promises API provides the following methods:

access, copyFile, open, read, write, rename, truncate, ftruncate, rmdir, fdatasync, fsync, mkdir, readdir, readlink, symlink, fstat, lstat, stat, link, unlink, fchmod, chmod, lchmod, lchown, fchown, chown, utimes, futimes, realpath, mkdtemp, writeFile, appendFile and readFile.

Update:

In Node 13 it works even with ESM modules:

import { promises as fs } from 'fs';

try {
    await fs.writeFile("/tmp/test6.js", "console.log('Hello world with Node.js v13 fs.promises!'");
    console.info("File created successfully with Node.js v13 fs.promises!");
} catch (error){
    console.error(error);
}

What is not right in this code? Post your solution.

Manuel Romero — Fri, 25 May 2018 03:41:23 +0000

const fs = require("fs");
const { promisify } = require("util");

const cache = new Map();

cache.set("file1", "data of file 1...");
cache.set("file2", "data of file 2...");

const readFilePromise = promisify(fs.readFile);

// What is wrong in this function?

const getFileData = (fileName, callback) => {
    if (cache.has(fileName)) return callback(null, cache.get(fileName));
    return readFilePromise(fileName)
        .then(data => {
            cache.set(fileName, data);
            return callback(null, data);
        }).catch(err => callback(err));
};

5 quick rules on modeling your MongoDB's schemas

Manuel Romero — Thu, 12 Apr 2018 04:10:29 +0000

NoSQL databases are a relatively recent technology and so that there are a lot of questions about when and how to use them.

I have read a lot of articles that say something like: "if your data has relationships, don't use NoSQL". That's not true! It's like a fake mantra that I've read over and over again. In the most of scenarios we will have to model data with any kind of relationship and NoSQL databases are ready to deal with it. The key is how we model our data.

I am not saying NoSQL databases are the best option if you have strongly related data, but having relationships in our data is not a sufficient argument to discard them.

We will focus on MongoDB. One on the most popular NoSQL databases (document oriented). IMO, part of its success is due to its easy integration with popular JavaScript Back-End frameworks. Moreover, MongoDB is the M in the popular MEAN and MERN stacks.

Below, I will give you 5 quick but powerful rules to take into account on designing MongoDB's schemas (you can use them for similar databases too):

As general rule, try to embed unless you have reasons not to do so.
If the objects you are going to embed may be accessed in a isolated way (it makes sense to access it out of the document context) you have a reason for not embedding.
If the array with embedded objects may grow in an unbounded way, you have another reason for not embedding. As a general rule, we should not embed more than a couple of hundreds of documents/objects neither more than a couple of thousands ObjectIDs.
Denormalize one or several fields of a document from the many side into the array in the one side (One-to-Many relationship) can save us some extra queries (application-level joins). IMO, denormalization is one of the keys to get the most out of this kind of databases. But, denormalization only makes sense if the denormalized field/s will be seldom updated. Otherwise, finding and updating all the instances is likely to overwhelm the savings that we get from denormalizing. So that, consider the read/write ratio for the field/s before denormalizing.
Don't be afraid of application-level joins. With the right indexing, they are barely more expensive than server-level joins.
(Update: last MongoDB versions includes the $lookup operator that allows us to do server-level JOINS, concretely LEFT OUTER JOINS).

And remember: Design your schemas the way through fit as well as possible your application's data access patterns. We want to structure our data to match the ways that our application queries and updates it.

If you want detailed information about how to model one-to-many and many-to-many relationships in MongoDB, check out my post: MongoDB schema design patterns (I)

References

MongoDB schema design patterns (I)

Manuel Romero — Thu, 29 Mar 2018 02:04:01 +0000

MongoDB has become one of the most popular noSQL databases. It is often used as a part of the MEAN/MERN stack because it is so easy to fit in the JavaScript ecosystem.
There are hundreds of tutorials on the Internet, tons of courses and some books about how to become a full-stack developer using MongoDB as the database system in the stack (The M in MERN/MEAN).
The problem is that the most of them don't focus on MongoDB schema design patterns. So that, operations/queries over designed schemas have so bad performance and/or don't scale.

One of the main problems you have to face on designing a MongoDB schema is how to model "One-to-N" (one to many) relationships.

Many beginners think that the only way to model “One-to-N” in MongoDB is to embed an array of sub-documents into the parent document, but that’s just not true. Just because you can embed a document, doesn’t mean you should embed a document. In fact, arrays that grows unbounded drops the performance. Moreover, the maximum document size is 16MB.

When designing a MongoDB schema, you must start with the question : what is the cardinality of the relationship? Is it “one-to-few”, “one-to-many”, or “one-to-squillions”? Depending on which one it is, you’d use a different format to model the relationship.

One-to-Few

An example of “one-to-few” might be the addresses for a person. This is a good use case for embedding – you’d put the addresses in an array inside of your Person object:

> db.person.findOne()
{
  name: 'Manuel Romero',
  ssn: '123-456-7890',
  addresses : [
     { street: '123 Sesame St', city: 'Anytown', cc: 'USA' },
     { street: '123 Avenue Q', city: 'New York', cc: 'USA' }
  ]
}

Pros:

The main advantage is that you don’t have to perform a separate query to get the embedded details.

Cons:

The main disadvantage is that you have no way of accessing the embedded details as stand-alone entities.

One-to-Many

An example of “one-to-many” might be parts for a product in a replacement parts ordering system. Each product may have up to several hundred replacement parts, but never more than a couple thousand or so. (All of those different-sized bolts, washers, and gaskets add up.) This is a good use case for referencing – you’d put the ObjectIDs of the parts in an array in product document.

Part document:

> db.parts.findOne()
{
    _id : ObjectID('AAAA'),
    partno : '123-aff-456',
    name : '#4 grommet',
    qty: 94,
    cost: 0.94,
    price: 3.99
}

Product document:

> db.products.findOne()
{
    name : 'left-handed smoke shifter',
    manufacturer : 'Acme Corp',
    catalog_number: 1234,
    parts : [     // array of references to Part documents
        ObjectID('AAAA...'),    // reference to the #4 grommet above
        ObjectID('F17C...'),    // reference to a different Part
        ObjectID('D2AA...'),
        // etc
    ]

Pros:

Each Part is a stand-alone document, so it’s easy to search them and update them independently.
This schema lets you have individual Parts used by multiple Products, so your One-to-N schema just became an N-to-N schema without any need for a join table!

Cons:

Having to perform a second query to get details about the Parts for a Product.

One-to-Many with Denornmalization

Imagine that a frequent operation over our Products collection is: given the name of a part, to query if that part exists for that product. With the approach that we have implemented we would have two do a couple of queries. One to get the ObjectIDs for all the parts of a product and a second one to get the names of the parts. But, if this is a common data access pattern of our application, we can denormalize the field name of the part into the array of products parts:

> db.products.findOne()
{
    name : 'left-handed smoke shifter',
    manufacturer : 'Acme Corp',
    catalog_number: 1234,
    parts : [
        {
         ObjectID('AAAA...'),
         name: '#4 grommet'
        },
        {
         ObjectID('F17C...'),    
         name: '#5 another part name'
        },
        {
         ObjectID('D2AA...'),
         name: '#3 another part name 2'
        }
        // etc
    ]

Pros:

We can see all the parts that belong to a product (its name) with one single query.

Cons:

Denornmalization makes sense when the denormalized field (name field in our case) is seldom updated. If we denormalize a field that is frequently updated, then the extra work of finding and updating all the instances is likely to overwhelm the savings that we get from denormalizing. A part's name will rarely change, so it's ok for us.

One-to-Squillions

An example of “one-to-squillions” might be an event logging system that collects log messages for different machines. Any given host could generate enough messages to overflow the 16 MB document size, even if all you stored in the array was the ObjectID. This is the classic use case for “parent-referencing” – you’d have a document for the host, and then store the ObjectID of the host in the documents for the log messages.

Host document:

> db.hosts.findOne()
{
    _id : ObjectID('AAA2...'),
    name : 'goofy.example.com',
    ipaddr : '127.66.66.66'
}

Message document:

>db.logmsg.findOne()
{
    time : ISODate("2014-03-28T09:42:41.382Z"),
    message : 'cpu is on fire!',
    host: ObjectID('AAA2...')       // Reference to the Host document
}

Conclusion

Based on the cardinality of our One-to-N relationship, we can pick one of the three basic One-to-N schema designs:

Embed the N side if the cardinality is one-to-few and there is no need to access the embedded object outside the context of the parent object.
Use an array of references to the N-side objects if the cardinality is one-to-many or if the N-side objects should stand alone for any reasons.
Use a reference to the One-side in the N-side objects if the cardinality is one-to-squillions.

And remember: how we model our data depends – entirely – on our particular application’s data access patterns. We want to structure our data to match the ways that our application queries and updates it.

Reference

What is overfitting in Machine Learning?

Manuel Romero — Fri, 09 Feb 2018 15:47:47 +0000

Overfitting is one the key problems you have to face when you come to Machine Learning world.

Theory says that overfitting happens when you have 100% or almost 100% prediction accuracy over your training set and when you check your model against unseen data (validation/test set), prediction accuracy is so far from training.

In other words, overfitting happens when your prediction model is too complex in relation with sample/data size/noise.

What does it mean? It means that your algorithm learned by heart the training instances of the sample.

So, why is it bad? Like in real life, when you have an History's exam and you just memorize (learn by heart) what the book says about American Civil War (for instance). If in the exam you are asked in the same way/order you have memorized the question, probably your mark will be 10/10. But if the question is something like "tell me about the social context of the American Civil War" you will say to your teacher: "This question was not in my History's book! The question was: American Civil War...that is all!!"Aha, as you learned by heart you are not able to answer this question.

Another example: How do we learn to multiply? Learning by heart the multiplication tables. But what happens when we have to multiply 12 x 45? There is no table for 12 neither 45. So, we shuold have learned that multiplying N x M is adding as N times the value of M.

Machine Learning algorithms work in a similar way. When it learns by heart, it is not able to generalize well. And that is the main goal, generalizing in the best way to make high accuracy predictions over new data (this new data, of course, rarely will be the same that training set data).

PD: Next post will be about how to solve overfitting.

Using ES6 array destructing with Promises

Manuel Romero — Thu, 09 Nov 2017 14:36:25 +0000

(originally posted more than one year ago on Medium)

As you know, some months ago, it was announced the new features of ES6 or EcmaScript2015. In the beginning not all of them were implemented by some browsers or Node.js, but we had tools like Babel.js to allow us to program in ES6 and then it would translate the code to ES5.
I am a Node.js developer and when I read the list of new features of ES6, there was one of them that could help me to continue fighting against callback hell and improve the readability of Promises. In particular, the readability of Promise.all() that, as we know, launches an array of promises in parallel. This feature is array destructing.

This is the way I worked with Promise.all() before using ES6 array destructing:

"use strict";

const getUserInfo = () =>
  new Promise((resolve, reject) => {
    resolve({
      id: 1,
      name: "Manuel",
      surname: "Romero"
    });
  });

const countUserFriends = () => new Promise((resolve, reject) => resolve(50));

const getUserSkills = () =>
  new Promise((resolve, reject) => {
    resolve({
      skills: ["JavaScript", "ES6", "MongoDB", "Angular.js"]
    });
  });

Promise.all([getUserInfo(), countUserFriends(), getUserSkills()])
  .then(results => {
    const userInfo = results[0];
    const numberOfFriends = results[1];
    const userSkills = results[2];

    console.log(userInfo);
    console.log(numberOfFriends);
    console.log(userSkills);
  })
  .catch(err => console.log(err));

In the code above, we have 3 promises that can be launched in parallel: the first is to get the user’s basic information. The second is to get the number of friends of the user and the last one to get the user’s skills.
As we can see, the variable that contains the result of the Promise.all() function is named results. Results is an array with so many positions as promises were sent as parameter to Promise.all(). Moreover, the result of the first promise goes in the first position of the array, the result of the second promise goes in the second position and so on.
But what I did to improve readability was assigning each result (position) of the array to a new variable. Because access to results[n] is not meaningful.
Now, let’s see how it can be done directly with ES6 array destructing:

"use strict";

const getUserInfo = () =>
  new Promise((resolve, reject) => {
    resolve({
      id: 1,
      name: "Manuel",
      surname: "Romero"
    });
  });

const countUserFriends = () => new Promise((resolve, reject) => resolve(50));

const getUserSkills = () =>
  new Promise((resolve, reject) => {
    resolve({
      skills: ["JavaScript", "ES6", "MongoDB", "Angular.js"]
    });
  });

Promise.all([getUserInfo(), countUserFriends(), getUserSkills()])
  .then(([userInfo, userFriends, userSkills]) => {
    console.log(userInfo);
    console.log(userFriends);
    console.log(userSkills);
  })
  .catch(err => console.log(err));

In the code above, I give a name for each position of the array. These names are the variables that we can manage in the callback. This way we save code and the most important thing, we write readable code easy to understand and easy to maintain.
PS: The two snippets (above) in this post were tested under Node v6.

Async/await version (extra for dev.to friends) :

"use strict";

const getUserInfo = () =>
  new Promise((resolve, reject) => {
    resolve({
      id: 1,
      name: "Manuel",
      surname: "Romero"
    });
  });

const countUserFriends = () => new Promise((resolve, reject) => resolve(50));

const getUserSkills = () =>
  new Promise((resolve, reject) => {
    resolve({
      skills: ["JavaScript", "ES6", "MongoDB", "Angular.js"]
    });
  });

const getAllUserData = async () => {
  try {
    const [userInfo, userFriends, userSkills] = await Promise.all([
      getUserInfo(),
      countUserFriends(),
      getUserSkills()
    ]);

    const userData = {
      userInfo,
      userFriends,
      userSkills
    };
    return userData;

  } catch (error) {
    return Promise.reject(error);
  }
};

/* MAIN */
getAllUserData()
  .then(userData => console.log(userData))
  .catch(error => console.log(error));

Liderazgo e Historia.

Manuel Romero — Sun, 24 Sep 2017 17:23:39 +0000

El otro dÃa, viendo la pelÃcula "300" en televisiÃ³n (no la habÃa visto aÃºn) e hilando con lo acaecido en Grecia posteriormente con Alejandro Magno (de los personajes que mÃ¡s me apasionan de la Historia) me di cuenta que se podÃa establecer una analogÃa entre los empleados y el jefe de una empresa y los soldados y el rey/caudillo militar de Ã©stos. Mirad: Tanto en el caso de la batalla de TermÃ³plilas o MaratÃ³n el ejÃ©rcito espartano (aunque realmente no eran 300, sino unos 7000 entre espartanos y soldados griegos de otras polis) derrotÃ³ en sucesivas ocasiones al ejÃ©rcito del Imperio Persa de Jerjes el Grande (rey de toda Asia) que era mucho mÃ¡s numeroso (unos 250000 hombres) y sofisticado que la tropa espartana + griega.

AÃ±os mÃ¡s tarde, Alejandro Magno, con un ejÃ©rcito de nuevo mucho mÃ¡s reducido que el del rival (DarÃo III, rey del Imperio Persa) liberÃ³ las polis romanas que habÃan tomado los persas, unificÃ³ Grecia bajo el reino de Macedonia y, lo mÃ¡s importante; conquistÃ³ en Oriente un territorio equivalente a 4 veces la extensiÃ³n de la PenÃnsula IbÃ©rica. Es mÃ¡s, una vez llegado a India decidiÃ³ volver a casa y terminar su campaÃ±a de conquista de Oriente debido a que sus soldados ya estaban cansados de luchar y de llevar varios aÃ±os fuera de su hogar, sin sus familias.

En ambos casos, un ejÃ©rcito pequeÃ±o pero bien organizado (Alejandro Magno pasÃ³ a la Historia como un gran estratega, al igual que su padre, Filipo II) derrotÃ³ no a uno, sino a varios ejÃ©rcitos en varias batallas mucho mÃ¡s numerosos y, a veces, con mejor tecnologÃa.

Â¿Por quÃ©? Os preguntarÃ©is... La soluciÃ³n es muy sencilla, tanto Alejandro Magno como LeÃ³nidas (rey de Esparta y defensor de Grecia en la pelÃcula 300) luchaban en primera lÃnea de batalla con sus soldados y, si habÃa que morir, se morÃa con ellos. Mientras que los reyes del Imperio Persa aguardaban en la retaguardia del campo de batalla mandando a sus soldados a Ã©sta a base de latigazos. Y si la cosa se ponÃa fea para su bando, simplemente huÃan.

Pues en una empresa pasa igual: si nuestro jefe lucha a diario con nosotros en primera lÃnea de combate, sus empleados tendrÃ¡n confianza en Ã©l, tendrÃ¡n seguridad en sÃ mismos, serÃ¡n valientes y productivos; no mirarÃ¡n atrÃ¡s porque saben que a su lado con la espada y el escudo estÃ¡ su jefe, luchando con y por ellos, como uno mÃ¡s.

Contacto: mrm8488@gmail.com | web personal