DEV Community: Dan Dascalescu

How to choose a vector database: Pinecone, Weaviate, MongoDB Atlas, SemaDB

Dan Dascalescu — Wed, 11 Sep 2024 14:09:15 +0000

While building a RAG application to search through my extensive public bookmarks collection (~15,000+ items collected since 2006), I looked in more or less detail at some of the available vector database solutions:

Pinecone - a SaaS offering that only stores vectors
Weaviate - an open-source vector DB with optional cloud hosting
SemaDB - a new entrant in the space, open-source
MongoDB Atlas - a mature document-oriented database that has added vector search in June 2023
Milvus - open source + cloud offering by Zilliz
Vespa, Qdrant, Chroma, Vald, FAISS (a vector search engine, not a database)

My requirements were flexible, but looking at the entire landscape helped narrow them down. When comparing a relatively large number of offerings, some choices become clearer, and the search space can be rapidly reduced. For example if two products seem largely similar but one doesn't have an important feature you (may) need, it's easy to pick the other one.

Thus, my requirements were refined to:

Open source, to avoid vendor lock-in. This disqualified Pinecone.
Single source of truth for the documents (bookmarks) and the vectors. This further disqualified Pinecone, which is centered around vectors and regards "metadata" as a secondary object associated with the vectors
Mature. This disqualified SemaDB, though it's a highly promising solution, and one of the only two that supports nested documents (the other one being MongoDB). Take a look at SemaDB's powerful search capabilities!
Notably, performance wasn't a goal, given the small dataset size

Also when comparing a large number of offerings, it's easy to dismiss products on more "superficial" grounds. For instance, Milvus presents with shoddy English in its documentation, and relatively scary complexity, and it's not clear what it would bring over Weaviate, so Weaviate wins.

That leaves us with Weaviate and MongoDB Atlas. It's unclear if Atlas vector search can be run on-prem, or if MongoDB's cloud offering must be used. I've asked their documentation AI this very question, but it replied that it doesn't know. It didn't even point to Creating local Atlas deployments. That page says "You should use local deployments for testing and development only."

While Atlas does have a free tier, Weaviate can be run on-prem, with a single binary even. Looks like we have a pretty clear winner.

Of course, Weaviate isn't without its problems. The TypeScript and Python clients aren't fully crystallized (both moved very recently moved to new major versions that deprecate the previous ones), documentation on running Weaviate on-prem without Docker is mysteriously lacking, and issues that have relatively simple fixes take a long time to get resolved.

The 5 settings to change in WebStorm right after installation

Dan Dascalescu — Fri, 02 Jun 2023 14:48:22 +0000

Map debugging keys to the shortcuts from Chrome, which can't be changed:
- Step Over: F10
- Step Into: F11
- Resume: F8
Editor -> Inlay Hints -> Position: Right, so you don't create a new line for the committer or number of usages of functions/methods
Disable reformatting on paste:
Editor -> General -> Smart Keys -> Reformat on paste: None
Editor -> General -> Tabs -> Mark modified (*)
Editor -> Code Style -> TypeScript: use single quotes
If you work with Markdown:
- Editor -> General -> Soft wraps -> Soft-wrap these files: add .mdx
- (New in WebStorm 2023.1.2) Disable Markdown table reformatting on typing (if you don't want to create large diffs when updating just a few characters in a table that's formatted differently from how WebStorm would like): Uncheck Editor -> General -> Smart Keys -> Markdown -> Reformat table when pasting

Understanding vector embeddings

Dan Dascalescu — Wed, 05 Apr 2023 08:59:07 +0000

Originally published on the Weaviate blog

Ever wondered how search engines find what you mean, not just the keywords you type?

Searching by the meaning of text, or by what objects are contained in images, is also called "semantic search". This is essentially a search by similarity, as opposed to keyword search.

For example, consider a library of wine names and descriptions, one of which mentioning that the wine is “good with fish”. A “wine for seafood” keyword search, or even a synonym search, won’t find that wine. A meaning-based search should understand that “fish” is similar to “seafood”, and “good with X” means the wine is “for X”—and should find the wine.

How can computers mimic our understanding of language, and similarities of words or paragraphs? To tackle this problem, semantic search uses at its core a data structure called vector embedding (or simply, vector or embedding), which is an array of numbers. A vector database like Weaviate (the startup I'm a Developer Advocate at) will compute an embedding for each data object as it is inserted or updated into the database. The embeddings are placed into an index, so that the database can quickly find the closest vectors to a given vector that will be computed for the query.

What exactly are vector embeddings?

Vectors are numeric representations of data that capture certain features of the data. For example, in the case of text data, “cat” and “kitty” have similar meaning, even though the words “cat” and “kitty” are very different if compared letter by letter. For semantic search to work effectively, representations of “cat” and “kitty” must sufficiently capture their semantic similarity. This is where vector representations are used, and why their derivation is so important.

In practice, vectors are arrays of real numbers, of a fixed length (typically from hundreds to thousands of elements), generated by machine learning models. The process of generating a vector for a data object is called vectorization. Weaviate generates vector embeddings using modules, and conveniently stores both objects and vectors in the same database. For example, vectorizing the two words above might result in:

cat = [1.5, -0.4, 7.2, 19.6, 3.1, ..., 20.2]
kitty = [1.5, -0.4, 7.2, 19.5, 3.2, ..., 20.8]

These two vectors have a very high similarity. In contrast, vectors for “banjo”, or “comedy” would not be very similar to either of these vectors. To this extent, vectors capture the semantic similarity of words.

Now that you’ve seen what vectors are, and that they can represent meaning to some extent, you might have further questions. For one, what does each number represent? That depends on the machine learning model that generated the vectors, and isn’t necessarily clear, at least in terms of our human conception of language and meaning. But we can sometimes gain a rough idea by correlating vectors to words with which we are familiar.

Vector-based representation of meaning caused quite a stir a few years back, with the revelation of mathematical operations between words. Perhaps the most famous result was that of

“king − man + woman ≈ queen”

It indicated that the difference between “king” and “man” was some sort of “royalty”, which was analogously and mathematically applicable to “queen” minus “woman”. Jay Alamar provided a helpful visualization around this equation. Several concepts (“woman”, “girl”, “boy” etc.) are vectorized into (represented by) an array of 50 numbers generated using the GloVe model. In vector terminology, the 50 numbers are called dimensions. The vectors are visualized using colors and arranged next to each word:

Credit: Jay Alamar

We can see that all words share a dark blue column in one of the dimensions (though we can’t quite tell what that represents), and the word “water” looks quite different from the rest, which makes sense given that the rest are people. Also, “girl” and “boy” look more similar to each other than to “king” and “queen” respectively, while “king” and “queen” look similar to each other as well.

So we can see that these vector embeddings of words align with our intuitive understanding of meaning. And even more amazingly, vector embeddings are not limited to representing meanings of words.

In fact, effective vector embeddings can be generated from any kind of data object. Text is the most common, followed by images, then audio (this is how Shazam recognizes songs based on a short and even noisy audio clip), but also time series data, 3D models, video, molecules etc. Embeddings are generated such that two objects with similar semantics will have vectors that are "close" to each other, i.e. that have a "small" distance between them in vector space. That distance can be calculated in multiple ways, one of the simplest being "The sum of the absolute differences between elements at position i in each vector" (recall that all vectors have the same fixed length).

Let's look at some numbers (no math, promise!) and illustrate with another text example:

Objects (data): words including cat, dog, apple, strawberry, building, car

Search query: fruit

A set of simplistic vector embeddings (with only 5 dimensions) for the objects and the query could look something like this:

Word	Vector embedding
`cat`	`[1.5, -0.4, 7.2, 19.6, 20.2]`
`dog`	`[1.7, -0.3, 6.9, 19.1, 21.1]`
`apple`	`[-5.2, 3.1, 0.2, 8.1, 3.5]`
`strawberry`	`[-4.9, 3.6, 0.9, 7.8, 3.6]`
`building`	`[60.1, -60.3, 10, -12.3, 9.2]`
`car`	`[81.6, -72.1, 16, -20.2, 102]`
Q: `fruit`	`[-5.1, 2.9, 0.8, 7.9, 3.1]`

If we look at each of the 5 elements of the vectors, we can see quickly that cat and dog are much closer than dog and apple (we don’t even need to calculate the distances). In the same way, fruit is much closer to apple and strawberry than to the other words, so those will be the top results of the “fruit” query.

But where do these numbers come from? That’s where the real magic is, and where advances in modern deep learning have made a huge impact.

How are vector embeddings generated?

The magic of vector search resides primarily in how the embeddings are generated for each entity and the query, and secondarily in how to efficiently search within very large datasets (see our “Why is Vector Search so Fast” article for the latter).

As we mentioned, vector embeddings can be generated for various media types such as text, images, audio and others. For text, vectorization techniques have evolved tremendously over the last decade, from the venerable word2vec (2013), to the state-of-the-art transformer models era, spurred by the release of BERT in 2018.

Word-level dense vector models (word2vec, GloVe, etc.)

word2vec is a family of model architectures that introduced the idea of “dense” vectors in language processing, in which all values are non-zero.

Word2vec in particular uses a neural network model to learn word associations from a large corpus of text (it was initially trained by Google with 100 billion words). It first creates a vocabulary from the corpus, then learns vector representations for the words, typically with 300 dimensions. Words found in similar contexts have vector representations that are close in vector space, but each word from the vocabulary has only one resulting word vector. Thus, the meaning of words can be quantified - “run” and “ran” are recognized as being far more similar than “run” and “coffee”, but words like “run” with multiple meanings have only one vector representation.

As the name suggests, word2vec is a word-level model and cannot by itself produce a vector to represent longer text such as sentences, paragraphs or documents. However, this can be done by aggregating vectors of constituent words, which is often done by incorporating weightings such that certain words are weighted more heavily than others.

However, word2vec still suffers from important limitations:

it doesn’t address words with multiple meanings (polysemantic): “run”, “set”, “go”, or “take” each have over 300 meanings (!)
it doesn’t address words with ambiguous meanings: “to consult” can be its own antonym, like many other words

Which takes us to the next, state-of-the-art, models.

Transformer models (BERT, ELMo, and others)

The current state-of-the-art models are based on what’s called a “transformer” architecture as introduced in this paper.

Transformer models such as BERT and its successors improve search accuracy, precision and recall by looking at every word’s context to create full contextual embeddings (though the exact mechanism of BERT’s success is not fully understood). Unlike word2vec embeddings which are context-agnostic, transformer-generated embeddings take the entire input text into account—each occurrence of a word has its own embedding that is modified by the surrounding text. These embeddings better reflect the polysemantic nature of words, which can only be disambiguated when they are considered in context.

Some of the potential downsides include:

increased compute requirements: fine-tuning transformer models is much slower (on the order of hours vs. minutes)
increased memory requirements: context-sensitivity greatly increases memory requirements, which often leads to limited possible input lengths

Despite these downsides, transformer models have been wildly successful. Countless text vectorizer models have proliferated over the recent past. Plus, many more vectorizer models exist for other data types such as audio, video and images, to name a few. Some models, such as CLIP, are capable of vectorizing multiple data types (images and text in this case) into one vector space, so that an image can be searched by its content using only text.

Vector embeddings with Weaviate

For this reason, Weaviate is configured to support many different vectorizer models and vectorizer service providers. You can even bring your own vectors, for example if you already have a vectorization pipeline available, or if none of the publicly available models are suitable for you.

For one, Weaviate supports using any Hugging Face models through our text2vec-hugginface module, so that you can choose one of the many sentence transformers published on Hugging Face. Or, you can use other very popular vectorization APIs such as OpenAI or Cohere through the text2vec-openai or text2vec-cohere modules. You can even run transformer models locally with text2vec-transformers, and modules such as multi2vec-clip can convert images and text to vectors using a CLIP model.

But they all perform the same core task—which is to represent the “meaning” of the original data as a set of numbers. And that’s why semantic search works so well.

Stay Connected

Thank you so much for reading! If you'd like to learn more about Weaviate, the open source vector database / semantic search engine, please check us out at weaviate.io!

The best sleep hygiene guidelines

Dan Dascalescu — Wed, 22 Sep 2021 14:30:48 +0000

Sleep is key for cognitive performance, which is key for being a successful software developer. Over the past 15 years, I've been consuming a lot of materials regarding sleep optimization, I've used some of the best consumer-grade sleep trackers (QS Emfit, Oura), and put together a summary of the best practices I have for optimizing sleep. 💤 Here we go: 🚀

Sleep Environment

No partner or pets in bed, or if a partner is a must, use an extra large bed and separate quilts.
Quiet bedroom; earplugs are great at reducing ambient noise. Some people prefer white noise machines.
Cool bedroom. The optimal sleep temperature is around 65F.
Dark bedroom. Blackout curtains work very well.
Dim lights in the evening.
This is less important than dimming lights, and more controversial, but see if it helps: reduce blue light exposure past sunset. Use apps/settings like "Night mode", or wear blue light blocking glasses.
Invest in a quality mattress, and sleep on the same bed, slightly harder than what you'd intuitively select. My favorites nowadays are the Helix mattress (tested) and Purple (not tested, but felt even better than Helix in store).

Timing

Consistent time to go to sleep AND wake up. Even on weekends. (Details on the circadian rhythm from Menno Henselmans and Andrew Huberman)
If you're trying to shift your schedule earlier (which may improve mental health and performance), do so in at most 30-minute increments (unless you're dead-tired). I've found myself many time trying to go to bed an hour or more earlier, only to be unable to fall asleep. Melatonin may (Gwern) or may now (Huberman) help or be indicated.
Bright light exposure as soon as possible after waking up, e.g. have breakfast by the window, walk outside. Note that Huberman cites research showing that glass reduces the effectiveness of light on circadian rhythm regulation by 50x (!). That means you'd need 10 minutes outside, or 500 minutes (a full workday) by the window to achieve the same benefits.
No snoozing. When it's time to wake up, wake up.
Avoid exercising 4 hours before sleep. [With the exception of sex]
Avoid stimulation before sleep, such as reading news, or having stressful interactions. Best to read a physical book (or an electronic one in dark mode) until you notice you slightly lose focus. Slightly boring books help.
Showering/bath 1-2 hours before sleep may help.

Food/supplements

Avoid caffeine 8 hours before sleep, or longer if you're particularly sensitive to it.
Avoid alcohol before sleep. It may help with falling asleep, but reduces sleep quality.
Ideally, avoid heavy meals several hours before sleep.
Carbs may help with sleep, so target them for dinner.
Experiment with melatonin, from 0.1 to 10g. Sensitivity is highly individual.
Avoid sleeping pills. They may help fall asleep faster, but the expense of quality.

Further resources

My summary of an excellent talk at Google
Art of Manliness on sleep (note that intermittent fasting may or may not work, by increasing cortisol)
Why 6 hours of sleep/night is surprisingly bad

Better sleep for programmers, according to Andrew Huberman

Dan Dascalescu — Wed, 22 Sep 2021 14:25:27 +0000

One controversial opinion in software is that your sleep quality and stress level matter far, far more than the languages you use or the practices you follow. Nothing else comes close: not type systems, not TDD, not formal methods, not ANYTHING.

This is my summary of Andrew Huberman's Master Your Sleep & Be More Alert When Awake | Huberman Lab Podcast #2.

My comments are [between square brackets]. Quotes are approximate.

TL;DR: view sunlight outside for 2 to 10 minutes in the morning, as early as possible, and before sunset. The spectral composition of low-angle sunlight controls the circadian rhythm much more than food or exercise. Avoid bright lights in the evening (or at least blue-ish lights), especially between 11pm and 4am. To fall asleep faster, practice Yoga Nidra, meditation, and sleep hypnosis, and use compounds if you need to: magnesium 300-400mg and theanine - 100-200mg 30-60m before sleep.

See my other post for the most effective sleep hygiene tips.

Salient quotes

"Sponsored by Helix mattress" [I slept on one for a moth, very good actually.]

"Most people wake up about when the sun rises, maybe 1 or 2 hours later" [Not us programmers...]

"Get 2-10 minutes of sunlight exposure regularly, as early in the morning as you can, outside. Early in the day, your retina is not as sensitive, so it needs a lot of sunlight photons to set clock mechanisms. Sunrise light intensity & color temperature (from sun being low in the sky) is ideal to set our circadian and hormonal rhythm. Outside light intensity is ~1000x higher than indoors."

"It's 50x less effective to view the low-angle morning sunlight through a window"

[THAT would be the paradigm shift for me. I've known about light as a trigger of the circadian rhythm for decades, and got it through the window, e.g. by having breakfast by the window. But if you need 10 minutes of direct sunlight, you'd need 500 minutes through glass, i.e. 8 hours by the window. Might happen in an office setting, or might not. As to how accurate the "50x" figure is, that's debatable.]

"The early morning light exposure needs to be done 2-3 days in a row for effects to manifest"

"Early in the day, the retina is not sufficiently sensitive to be disrupted by light, so it's OK to look at your phone if you wake up before sunrise; you need to look at the sun for longer if it's overcast"

"Blue blockers should be reserved for the evening (after 8pm), or if you wake up in the middle of the night and use screens (@53:15)"
[That was contradicted by this study on mice, which suggested that warm yellow light disrupts sleep worse than blue]

"Viewing low-angle light when the Sun is setting can help protect the circadian mechanisms against the negative effects of light later in the day"

[At 46 minutes in, he missed the opportunity to talk about sleep masks. Seems they're useless, because the eyes wouldn't see light while closed. I doubt that. You can still see strong light with your eyes closed.]

~"The longer you've been awake, the more sensitive the retina becomes, so even small amounts of light can disrupt your sleep patterns. Get as much light as safely possible before sunset, and as little after ~8pm; and no bright light exposure 11pm-4am"

~"Light that arrives in the eyes between ~11pm-4am suppresses the release of dopamine (endogenous antidepressant, learning facilitator)"

"Red light won't trigger these issues"

"The cells in our eyes that signal the central clock reside mostly in the bottom half of our retina which means it'€™s viewing our upper visual field. To avoid improper activation of neurons, place light low in your physical environment, on desktops or floor." [I don't like lights that low, you inevitably look at them and get retinal afterburn]

"Candlelight and fireplaces are fine, very dim lights are fine"

"Shift work needs an entire discussion" [how about us PARTY GOERS?!]

@55:40 "If you turn on the lights ~45-60 minutes, even if your eyelisds are closed (provided you're not under the covers), that increases your total sleep time and makes you want to go to bed earlier each night" [this suggests sleep masks do make a difference]

"So the light does penetrate the eyelids and activates the neurons that control the central clock"

"You can advance or delay your sleep/wakefulness phase. If you see light late in the day, esp. in the middle of the night, your brain will think that's morning light, even though it's not sunlight, and will delay your clock, making you want to get up later and go to sleep later"

"If you wake up early (4-6-7am) and get light exposure, it will phase-advance your clock"

"..people's internal mechanisms aren't anchored to anything regular"

"If you can provide them consistent light anchors early in the day, and in the evening, and avoiding light at night, you'll be amazed at the tremendous positive effects - hormones, wellbeing"

"Most people are not familiar with what it is [like] to sleep really really well on a consistent basis"

"Naps of ~20-30m (up to one hour) can be beneficial. If you wake up from a nap feeling groggy, that means you didn't sleep well enough during the night"

"Non-Sleep Deep Rest: meditation, Yoganidra, hypnosis"

"NSDR can help with emotional stability, wakefulness, and falling asleep more easily at night"

Supplements

modafinil for narcolepsy
adderall for wakefulness makes you feel more alert, there tends to be a heavy rebound, and there's addiction potential
magnesium - increases GABA, makes mind drift in space & time. Magnesium threonate is the best for inducing drowsiness, 300-400mg 30-60m before sleep
theanine - 100-200mg 30-60m before sleep
50mg of apigenin can support falling and staying asleep. It's an estrogen inhibitor; esp. important for women"

How to fall asleep

"Yoganidra - meditation you listen to that allows you to consciously bring your body to a state of deep relaxation; sometimes you fall asleep. It involves passively listening for 10-30 minutes to a script about body scanning and relaxation"

Sleep hypnosis - Reveri by colleague David Spiegel [I tried it once when I couldn't fall asleep, and it didn't work]

"It's very hard to control the mind with the mind. When you have trouble falling asleep, you need to work with the body to control the mind"

NB: try also this 4-7-8 breathing technique

How to extract your Facebook Messenger messages from a conversation

Dan Dascalescu — Wed, 22 Sep 2021 13:36:29 +0000

If you communicate with people (or bots) over FB messenger, you may find it useful to save a copy of your messages. You can do this by copy/pasting, but the output will be littered with "You sent", scrolling up is a pain, and you can't filter for only what you said.

Fortunately, Facebook provides data exports in JSON format, which we can then filter easily with the jq utility.

Go ahead, Request a Copy in the date range you care about, and select only "Messages":

Then extract from the archive the file messages/inbox/<your_friend>/message_1.json. Once you have that file, here's how to filter all your messages, sorted chronologically by time:

cat message_1.json | jq -rc 'sort_by(.timestamp_ms) | .[] | select(.sender_name | contains("Dascalescu")) | .content'

(of course, replace "Dascalescu" with your name)

The -r flag unquotes the strings. Useful to unescape "s.

That's it!

Known issues

Characters are output as raw bytes. This will break non-ASCII characters and emojis. Check for solutions on StackOVerflow.

How to unregister service workers in Firefox

Dan Dascalescu — Sat, 08 May 2021 06:42:22 +0000

Just a quick post because the Google search results for this are out of date.

To stop (unregister) service workers in Firefox, go to

about:debugging#/runtime/this-firefox

Unfortunately there's no way to filter (as there is for cookies), so if you've been running that Firefox instance for a while, you'll likely have tens or hundreds of registered service workers.

Keep malicious Node modules behind bars!

Dan Dascalescu — Mon, 12 Apr 2021 21:19:18 +0000

When you use a Node module, have you thought that maybe a dependency of a dependency that your script uses, could become malicious?

That dependency has the same access to your computer as your own code. It could look through your files then phone home anything it wants, as recently seen with crypto wallets in the event-stream attack, or sensitive Linux files. These types of attacks are called "open source supply chain attacks", and hundreds of malicious package attacks have been documented.

Deno forbids access to the filesystem by default, and you can whitelist specific directories. Node doesn't have this capability.

One solution is to run your scripts in a sandboxed environment such as firejail, bubblewrap, or a VM (Docker is a containerization solution, not aimed at security).

Once you've installed and configured sandboxing, it would help to check if your scripts are indeed sandboxed correctly, and abort immediately otherwise, so that no dependency has a chance to execute. To that purpose, I've published a module called "behind-bars", which does exactly that. All you have to do is add this line at the top of your script:

import 'behind-bars';

The module will call process.exit() immediately if it can access sensitive files or directories (browser profiles, cryptocurrency wallets, ~/*_history etc). It does so before any other imported module has a chance to run and steal data, by only using synchronous calls. You can optionally configure it to also ensure there's no Internet access, and you can define your own custom paths to sensitive files.

So far it checks for common sensitive files on Linux and MacOS systems. PRs for Windows are welcome in the repo!

Implementation challenges

To provide the maximum level of security, the module code needs to execute before any other imported code. Thus the developer must add the import 'behind-bars' line first at the top of the script. At that point, the module code needs to complete two types of requests before any other code runs:

Checking the filesystem for accessible paths
Checking for Internet access (by fetching a specified URL)

Both these requests are normally asynchronous, which means other module code will get executed before they complete. This is unacceptable. Starting with Node v14.3.0, top-level await has been unflagged, and we could use it to wait for the requests to complete. The problem is, when transpiling to CommonJS, top-level await doesn't actually block the other imports from executing.

This leaves using synchronous requests as the only backwards-compatible option. Unfortunately this means using old libraries like glob and sync-requests, which pull a total of ~40+ dependencies. If dropping CommonJS compatibility were ok, we could use the far lighter tiny-glob, and node's native http/https module.

Proud to have reached 100k rep on StackOverflow

Dan Dascalescu — Wed, 23 Dec 2020 19:50:43 +0000

I haven't been active on SO lately, and I randomly found this old email in my inbox today - turns out indeed that I reached 100k rep, and almost 20 million people since I joined in 2009.

Despite the more than occasional frustration, StackOverflow has been the most useful community resource in my career as a software engineer, and I'm deeply grateful for its existence.

The most revolutionary software probably being developed right now

Dan Dascalescu — Wed, 14 Oct 2020 01:18:46 +0000

NOTE: I initially wrote this post before I was aware of GPT-3. I did somewhat anticipate GPT-3 (see the "rhetoric" bullet below), but its applications may be truly revolutionary in a few years. See Awesome GPT-3.

This post was inspired by a previous discussion on the topic, that received a surprising number of very narrow examples: some CSS library, some JS framework, some project management app.

By "revolutionary", I'm thinking about software with the potential to profoundly impact life. This can be in established fields, or in emerging ones. I'm pretty sure there are efforts underway to build each of the examples below; we may just not know about them yet. Here are some that I know of, or have imagined:

machine learning - autoML
law - deep-learning digital attorney able to find the most appropriate legal precedent in a matter of minutes, vastly outsmarting human lawyers
politics - liquid democracy, or other voting systems solving the one-person-one-vote problem
rhetoric - argument-making software able to persuade anyone on any controversial topic with a mix of flawless logic, appeal to emotion, and plain abuse of reasoning fallacies. This can have immense implications on politics, far beyond what fake news and social media manipulations have had, even at the current state of technology. UPDATE post GPT-3 and Neuralink: imagine a leader whose brain is connected to Neuralink and can come up with a brilliant, convincing, fact-checked, (counter)argument to anything, better than Gary Shapiro and the Limitless protagonist. Society is not ready to openly accept AI-based governance, even though we are already at the mercy of various algorithms deciding loan interest rates and whatnot. In 20-50 years, a charismatic leader with hidden wireless access to the successor to several generations of GPT-3 like algorithms, can plausibly take over the world.
molecular biology - figuring out protein folding
blockchain - encrypted storage of surveillance data (CCTV footage, Internet traffic etc.), whose decryption can only be done with a combination of keys from the owner of the data source and the authorities, and such decryption is permanently marked and visible in the blockchain. This would legitimize surveillance because the use of the data would be revealed.
prediction markets - Augur looks promising
SAFE Network - decentralized, autonomous and encrypted web
transportation - cheap self-driving modules
weather - high-accuracy weather prediction compensating for chaotic system effects
BCIs - intepreting subvocalizations for silent computing, AI software to decode brain waves in general
synthetic biology - iGEM, GenoCAD, and other tools

I'm sure there are a lot more initiatives that can significantly change life as we know it. Please drop them in the comments below. After all, software has been eating the world for a decade now.

Modern TypeScript project template

Dan Dascalescu — Mon, 27 Jul 2020 03:00:47 +0000

Now that ES Modules are native to Node 14, it's time to make TypeScript generate ES Modules code, even though it doesn't support the .mjs extension yet.

Goals

emit modern ES modules code
import modules that use Node built-ins
import modules that don't have named exports (e.g. apollo-server, node-influx)
import your own modules without specifying an extension
lint with ESLint, with TypeScript support
test the TypeScript code instantly without having to build first
run the resulting JavaScript code, with support for the optional chaining operator ?.

Bonus: continuous integration script for GitHub Actions. It automatically runs tests on every pushed commit.

Bonus points: don't use Babel!

OK, here's how to do this. We'll start with a new Node project (npm init) and a standard tsconfig.json file, then highlight the important changes.

You can see the complete template repo at

https://github.com/dandv/typescript-modern-project

Emit ES modules code

In tsconfig.json, set this under compilerOptions:

    "target": "esnext",
    "module": "esnext",  // Output `import`/`export` ES modules

Import modules that use Node built-ins (`http`, `url` etc.)

run npm install --save-dev @types/node
in tsconfig.json under compilerOptions, set
- "moduleResolution": "node", so tsc can find modules when targeting ES6+
- "types": ["node"] to avoid errors related to Node built-in modules

Import modules that don't have named exports

Normally we could write in TypeScript

import { InfluxDB } from 'influx';

but when generating ES modules code, that statement will be passed through as is, and will cause Node to fail with

SyntaxError: The requested module 'influx' does not provide an export named 'InfluxDB'

because node-influx doesn't provide named exports (and neither does an even more popular module, apollo-server).

One alternative would be to generate old ugly commonjs modules code by,

removing the "type": "module" line from package.json, and
changing the module line to "module": "CommonJS" in tsconfig.json (allowSyntheticDefaultImports also becomes unnecessary)

What we'll do is import the entire module:

import Influx from 'influx';
const influx = new Influx.InfluxDB();

However, this will generate Error TS1192: Module '...' has no default export. To prevent that, set "allowSyntheticDefaultImports": true in tsconfig.json.

Import your own modules without specifying an extension

When transpiling, TypeScript won't generate an extension for you. Run Node with the node --experimental-specifier-resolution=node parameter:

node --experimental-specifier-resolution=node run.js

Otherwise, node mandates that you specify the extension in the import statement.

To support optional chaining, add the --harmony flag to the node command line.

Run the resulting JavaScript code

Add "type": "module" to package.json, because TypeScript can't generate files with the .mjs extension.

ESLint

To be able to run eslint, we must create an .eslintrc.cjs file, rather than a .js one (due to "type": "module" in package.json). Then, install the required dependencies:

npm i -D eslint @typescript-eslint/eslint-plugin @typescript-eslint/parser

Here's the diff to add ESLint support.

Testing with Jest

The cleanest way to fully support Jest with TypeScript, ES Modules and ESLint, is to use ts-jest, which has a number of advantages over using Babel. What we need to do:

npm install --save-dev jest @types/jest eslint-plugin-jest - for ES Lint support
add "jest" to the types array in tsconfig.json
add the 'jest' plugin to .eslintrc.cjs and also add 'jest/globals': true to its env key
use the jest.config.cjs generated by ts-jest config:init (just renamed .js -> .cjs).

Normally, to run Jest from package.json, we'd add a "test": "jest" line. That won't be sufficient, because we need to pass the --harmony flag to node (for optional chaining support).
To pass parameters to Node when running Jest, we'll use the following test line:

"test": "node --harmony node_modules/.bin/jest"

The only caveat here is that Jest seems to prefer generated .js files over their .ts originals, so we'll exclude them via jest.config.cjs:

  testRegex: '.*.test.ts',  // test filenames matching this regex
  moduleFileExtensions: ['ts', 'js'],  // modules are only in .ts files, but 'js' *must* be specified too

Source maps

If your script generates an error, you'll see the line numbers from the generated .js files, which is not helpful. We want to see the original paths and line numbers from the .ts files. To do that, we'll add sourceMap: true to tsconfig.json, install source-map-support and run node with the -r source-map-support/register parameter. Note that Jest already takes care of source mapping so you'll see the .ts line numbers without having to do anything extra.

Here's the diff to add source map support.

CI testing

Using GitHub Actions, we can configure automatic testing via .yml files under .github/workflows. You'll see a ✔️ in the repo when tests pass.

TODO

The tsconfig.json settings generate .js built files, and .js.map source map files, next to the original .ts file. While some IDEs conveniently hide these files, it may be desirable to output them in a separate directory, typically dist.

This can be done using the rootDir/outDir settings in tsconfig.json, but that sort of setup comes with an annoying limitation of Typescript that forbids importing files outside the rootDir. That can be a problem with monorepos.

Wrap up

That's what you need to do, in order to get modern TypeScript code to produce modern ES Modules. Quite a lot of stuff. Thankfully, Deno compiles TypeScript natively, so I'm looking forward to those bright days 🌞

9 problems with replacing "master" in Git

Dan Dascalescu — Mon, 15 Jun 2020 22:27:46 +0000

[Background for those unfamiliar with this debate: in summer 2020, as if the world didn't have enough real problems (Covid-19, the George Floyd protests), some activists craving social signalling, started to argue that the word "master" must be cancelled in all areas of human endeavor, because at some old point in time, it referred to the master of black slaves. One of those areas is software engineering, where the word master never had any association with slavery, but simply meant "primary", as opposed to "secondary". Why this particular war was started now, and not a continuation of the one against Master's degrees, I don't know.]

This post started as a comment reply to "Replacing master in git" but it turned into a longer topic. The line that sparked my reaction was this:

"Master/slave is an oppressive metaphor"

There so many problems with that statement. First off...

1. There's no "slave" in Git

Maybe MongoDB has this problem, but not git. So there's no "master/slave" metaphor in Git to speak of.

2. Words change meanings over time

"Master" is a term used the recording industry, or to talk about skill ("Master of Science", "Kung Fu Master" etc.). I'm not a native English speaker so "master" was strange to me too, but in American English "master" has been far more commonly used in the past 50 years to talk about vinyl records and CDs and degrees and expertise, than slavery. As someone else said,

These terms have existed for decades and now suddenly we feel they're racially loaded? I think this is a knee jerk reaction to a problem that doesn't exist.

3. Nothing is oppressive by itself

People choose meanings. Epictetus famously said this of insults (and "master" is far from an insult, but to illustrate the point):

“Remember, it is not enough to be insulted to be harmed, you must believe that you are being harmed. If someone succeeds in provoking you, realize that your mind is complicit in the provocation. It is not he who reviles you who insults you, but your opinion that these things are insulting."

4. Who does the word "master" in Git really offend?

The vast majority of developers have nothing to do with oppression, and have never intended to oppress anyone.

Has anyone ever complained of being offended when they saw a "master" branch in a Git repo? From what I've seen so far, black developers haven't felt offended by this word.

Beautus S Gumede • Jun 15 '20 • Edited

I think it's a good gesture. But it feels a bit awkward as a person of color to see such things when I've never felt that way as from the time I knew about computer science and to where I am now. This feeling though is influenced by many things, partly being that I'm from a different place where some terms aren't as heavy hitting in terms of my history and that I cannot speak for other people of color from other locations in the world.

Re-purposing has more impact than renaming, in my opinion. Changing a branch name from 'master' to 'default' carries less weight compared to github just not working with government institutions like Immigration and Customs Enforcement's ICE. Removing "blacklist/whitelist" in favour of something which is probably more harder to remember has less impact than not blocking open source accounts and repositories from locations like Iran.

Historical names do way less harm than the inhumane practices corporations do today. But that's just my opinion

Olivier JM Maniraho • Jun 16 '20

I totally agree with your points, I think these kind of changes are only making things worse than they were before, I never heard anyone who ever complained about this term.

I am a developer and I am black(not speaking for anyone else of color) yet I have never felt offended by a word I see in a software that I use or don't use, I think there are far much better things we can do as a community if we agree to live with our different cultures.

Thanks to Ben Calder for pointing out that some Drupal contributors felt offended by master/slave. That's different from master in the Git world.

If nobody is actually choosing to be offended by this word, then all we're doing is wasting time changing a perfectly working process just to virtue-signal that we're "woke", while annoying far more people in the process and wasting their time when "master" will no longer work and its replacement will break things.

5. What are the downsides of this rename?

When making a decision of this scale, we need to realize that it invalidates 15 years of scripts and build pipelines and CI processes and Git tutorials using the word "master". What is its ROI? That's a legitimate question. We think "master" may offend the sensibilities of some black developers, and OK, there might be some I haven't seen yet, but what about the actual pain inflicted on those who have to deal with the change? Broken builds, staying up al night fixing a P0 caused by the change, getting fired because you haven't exhaustively tested before deploying, downtime caused to customers who have no fault in this - these are real pains the change will cause, as any DevOps person will tell you.

6. Is this purely a PR move?

By being outspoken against "master", brands benefit from stoking the fires of racial conflict to elevate themselves in the eyes of consumers. Sony, an organization using cobalt mined by literal black slaves, including children, is in favor of BLM. GitHub renaming a branch seems like a cheap PR stunt by comparison, when they could do far more meaningful things, like not working with the ICE:

...or not banning Iranian developers, suggesting they use GitHub to develop nuclear weapons.

7. What shall we rename next?

Slaves were hanged from branches at some point, should the word "branch" be changed? How about the Lighthouse web speed auditing tool? Light is usually white... Going forward, we could realistically decide to rename "Master Degree"s.

I know this is a slippery slope. We've been on it! In 2015, the file extension ".bro" was deemed offensive was deemed offensive and had to be changed to .br. In various cultures (Australia, New Zealand), "bro" is a very positive term. But one irritated person was offended. The point of the slippery slope is, where will it stop? I bet whoever found "bro" offensive didn't stop at that.

Have we, by changing the extension, helped that person be more tolerant, inclusive, resilient, calm, or in any way better than before?

I wonder how that person reacted to this parody:

ErisBlastar / cplusequality

Feminist Software Foundation C+=, a new language for us feminists

A project of the Feminist Software Foundation. Feminist software is a cornerstone of any modern free society. We build this foundation.

Trigger Warning, this repository uses satire and sarcasm and other types of humor that could trigger Feminazis into rage fits. Maintainer of this repository was gone for a while due to being source code raped by feminists who lack a sense of humor. Which is ironic as this language was written for feminists by women. We are trying to help out women and feminists to have a programming language not controled by the capitalist patricarchy and men.

Now hosted on BitBucket, as GitHub proves to be too misogynistic to support a feminist programming language.

C+=

C+= (pronounced either C-plus-Equality, or See Equality) is a feminist programming language, created to smash the toxic Patriarchy that is inherent in and that permeates all current computer programming languages.

…

View on GitHub

Anyway, odd fact about me: I actively dislike being called "bro". It's irrational, and I have no explanation for it. But I don't mind it when my Aussie friends call me "bro". They don't mean anything bad by that; on the contrary, "bro" is a term of fraternity, of endearment. Someone being offended by a file extension, which doesn't mean anything but short for the compression algorithm "Brotli", is quite puzzling to me.

A slippery slope is a fallacy when it's used to support an argument. Here, it's merely a fact. I'm not arguing we should not rename "master". Go right ahead, rename it. I'm just pointing out some problems with getting up in arms about it.

8. Don't we have better things to worry about?

Could the time collectively spent on this debate have been better spent in other ways? Or do we like the illusion of doing something useful? I think that arguing about how we think black people (should?) feel about a word takes attention away from real issues.

9. Is this humiliating to black developers?

SK (Panda Global)

@speedkicks

Reading a thread of white people, including the CEO of GitHub, advocating changing the name of the "Master" branch to make black devs more comfortable...

is the most racially uncomfortable I've ever felt about GitHub.

21:44 PM - 14 Jun 2020

Last, and worst, by removing the word "master" blindly, we're uncritically reproducing a narrative that diminishes black agency in favor of a white-centric explanation. We're making the assumption that a black developer can't take the word "master" as simply a label and nothing more. Don't you think black developers may find that assumption humiliating?

PS: I've looked at reactions to similar moves in the past. The founder of Ruby on Rails pontificated about renaming master back in August 2018. The reaction of the developer community was pretty much the same as mine.