DEV Community: Nic

My Favorite Tech Blogs

Nic — Thu, 30 Dec 2021 00:09:30 +0000

Hope you’re all enjoying your holidays!

Today I want to share some of my favorites tech blogs. I have followed these sites for many years, and I think they are worth your time.

The criteria I recommend are:

Continuous updates for many years
High quality and some articles have inspired me deeply

Eli Bendersky’s website

I remember I found this blog when I tried to finish all the SICP exercises in 2008. There are almost all the answers to SICP exercises with detailed explanations!

What is commendable is that the author Eli Bendersky has always insisted on writing his technical experience, and this blog is continuously updated. The quality of the content is very high, with both theory and code. For instance, he wrote the Raft series: Implementing Raft: Part 0-Introduction, which is simply a model of technical writing.

Flavio Copes

This blog is maintained by an Italian developer, and the content is mostly on front-end and web development.

I admire this blog because he keeps blogging almost every day! What he wrote may not be in depth, but it is useful to many people. I used SEO tools to read the data of this website, and the organic traffic brought by Google is very high.

I recommend everyone who want to start with writing to read these two articles:

Every developer should have a blog. Here’s why, and how to stick with it

I wrote 1 blog post every day for 2 years. Here’s 5 things I learned about SEO

Julia Evans

This blog covering a wide range of fields. This author deeply inspired me to learn programming by understanding internal principles: Get better at programming by learning how things work . Of course there are many other technical articles such as: Diving into concurrency: trying out mutexes and atomics

And the author has made a lot of fancy e-books: wizard zines

journal.stuffwithstuff.com

He wrote a book on programming language implementation in 6 years, with the spirit of craftsmanship: Crafting “Crafting Interpreters”

All the pictures here are drawn by hand, all details on font, color, alignment, etc., all these details are almost perfect, and the final book is an artwork. A technical book can be so beautiful!

swyx’s site

The content involves development, personal growth, etc.

Among them, the concept of Learn In Public (swyx.io) touched me a lot, and there are many awesome podcasts on this blog.

That all for today, if you like it please tell me, I actually have more to share with you. 🙌

See you!

How to Keep A Program Running with Ssh logout

Nic — Thu, 25 Nov 2021 15:08:01 +0000

Some times we need to keep jobs running without a ssh session, here there are three ways to achieve this. I suggest to learn a terminal multiplexer in long term, it’s really useful when you have long time working on Linux/Unix system.

Option 1: `nohup`

The best and simplest way is:

nohup long-running-command &

It was made specifically for this, by default it will log stdout to nohup.log, if standard output is not a terminal, logs will be appended to ~/nohup.log. If you want to specify the output log into a different file:

nohup COMMAND > FILE

More details can refer to man nohup. What nohup does, is to effectively separate the process from the terminal:

It closes standard input (the program will not be able to read any input, even if it is run in the foreground. it is not halted, but will receive an error code or EOF).
It redirects standard output and standard error to the file nohup.out, so the program won’t fail for writing to standard output if the terminal fails, so whatever the process writes is not lost.
It prevents the process from receiving a SIGHUP (thus the name).

Option 2: `bg` + `disown`

If you already have started the program and don’t want restart it, we can use ctrl+z to stop current execution and then use bg to resume it and switch the running program to background (We can use fg to pull the job to frontground)

Finally we use disown to remove the process from the shell’s job control.

So, even you logout from a ssh session, the program will still keep running in background.

ctrl+zbgdisown -h

Option 3: Use terminal multiplexer

We have three choices for this, [[screen]], [[tmux]], and [[byobu]].

screen and tmux is mostly used ones. On Ubuntu or Debian system, we install it with:

sudo apt-get install screen sudo apt-get install tmux

Byobu is an enhancement for the GNU Screen, you can install it with this commands:

sudo add-apt-repository ppa:byobu/ppasudo apt-get updatesudo apt-get install byobu

Rust Warp: Use Cookie for Authorization

Nic — Wed, 24 Nov 2021 13:11:48 +0000

Warp is a composable, web server in Rust. It's code is very small and focus on speed.

The fundamental building block of warp is the Filter, they can be combined and composed to express rich requirements on requests.

But it's actually not easy to use if you are not familiar with it's concept, and the type system will also scare some beginners.

For example, I actually spend some time to figure out how to use Cookie for authorization.

Suppose we have defined a Struct to represent the User:


#[derive(Debug, Deserialize)]
pub struct User {
    username: String,
    password: String,
}

And we have a logic to authorize whether the login request has a valid username and password, the detail implementation depends on your code:

pub fn verify_user(user: &User) -> bool {
    ....
}

The question is how to set a Cookie in response and how to verify each request after authorization.

Set a cookie

When a user send login request and passed the authorization, we use reply with with_header to set a cookie to store a token, which will be used for later requests:

 let login = warp::path!("api" / "login")
        .and(warp::post())
        .and(warp::body::json())
        .map(|user: auth::User| {
            if auth::verify_user(&user) {
                let token = auth::gen_token();
                warp::reply::with_header(
                    token.clone(),
                    "set-cookie",
                    format!("token={}; Path=/; HttpOnly; Max-Age=1209600", token),
                )
                .into_response()
            } else {
                warp::reply::with_status("failed", http::StatusCode::UNAUTHORIZED).into_response()
            }
        });
    let routes = routes.or(login);

Use cookie for Authorization

To authorize request, we need to implement a filter in Warp, and use it like this, here we use Filter::untuple_one to unroll nested tuple layers from extractions.

  let verify = warp::path!("api" / "verify")
        .and(warp::get())
        .and(auth_validation())
        .untuple_one()
        .map(|| warp::reply::reply().into_response());
    let routes = routes.or(verify);

And the auth_validation will call another built-in filter warp::cookie to extract the token from request header:

struct Unauthorized;

impl reject::Reject for Unauthorized {}

fn auth_validation() -> impl Filter<Extract = ((),), Error = Rejection> + Copy {
    warp::cookie::<String>("token").and_then(|token: String| async move {
        println!("token: {}", token);
        if let Some(true) = auth::verify_token(&token) {
            Ok(())
        } else {
            Err(warp::reject::custom(Unauthorized))
        }
    })
}

Some thoughts

Even I have spent some time writing code in Rust, I still need to learn some new concepts or some details in a Web framework such as Warp. From my experience, there are some unnatural part for users who used to some other programming languages. For example, the return type of a filter:

impl Filter<Extract = ((),), Error = Rejection> + Copy

It's just not so easy to say what this mean? And why we need to call untuple_one with the filter?

I know we must obey the rules of type system, and we need to add more annotations when writing Rust code, it's just not so easy for learning as other programming languages.

Even so, I'm still having fun with it for it bringing some new things for programming.

The Best Way To Learn Frontend As An Experienced Programmer

Nic — Tue, 23 Nov 2021 15:33:36 +0000

I have done some front-end programming recently. Even I have been coding for almost 16 years, I haven’t done much on the front-end in my coding time.

The reason is, I never found Front-end programming interesting. I thought front-end is only manipulate the HTML elements, and provides some styles for the UI of web. I found CSS is trivial and there is not logic behind the css code. We must remember a lot of stuff to write CSS. I don’t want to remember …

Until recently, I have done more work in front-end. My career don’t involve front-end much , but I have spent some time to learn it by myself. The motivation behind it is my side-project.

I finished several projects in my spare time, and got more interests on it. Here are some of my projects:

gomoku
obweb

So, with about one year experiences of practice, I improved much on it. I found these rules for learning front-end as an experienced programmer. It also may help for beginners:

Learn by doing

This is always my #1 role of learning programming.

I guess I still won’t have learned frontend programming without my side-project. I have some desire on my side-project, I tried to finish it with perfection in my mind.

Learning frontend is the extra credits of finishing my projects. So, if you don’t have much desire on learning frontend(or programming), maybe you need to find some thing which interested you, try to find those things attract you desire. For instance, if you want to develop a game, you may need to learn Game programming or something like C++ programming.

When you begin to get hands dirty with your keyboard, other things will follow.

If you don’t find any good project to start with, here are some learn-by-doing style of courses:

Learn to Code (freecodecamp.org)

Play with open-source projects

It’s the best time for learning programming right now. Compared with 10 years ago, we have many many open-source projects to learn.

As as a beginner in front-end, I get much fun and interests when I played all the demo of this project: Mini projects built with HTML5, CSS & JavaScript..

I read the source code of each sub-project, and try to do some modification on them. After finish it, I learned a lot.

Read Books, Ignore Videos

When you want to go deeper about front-end, there are actually a lot of theory stuff need to learn. For these essential knowledge, I would suggest you to learn by reading some classic books, such as:

There are many videos for learning Web development, it’s only helpful when you are starting without any experience. If you have got some learning experience, you should step out relay on watching those videos. Because almost none of them will teach you deeper skills and knowledges. Classic books and other authoritative materials will be helpful.

The Great Programming Quotes

Nic — Sat, 10 Jul 2021 06:11:37 +0000

Several days ago, I came across John Carmack’s post on learning programming. His advice is truly helpful for programming beginners and worth more reading.

This reminds me to spread other great quotes in mind, which from great programmers and computer scientist.

Some of them help me understanding more on computing, some of them are principles I’m trying to apply to my daily work, and some of them are funny.

Design and architecture

Abstraction is essential

Complexity is anything that makes software hard to understand or to modify.

-–John Ousterhout

“All problems in computer science can be solved by another level of indirection”

– David Wheeler

The power of these can be seen in the domains of software development, design patterns, architecture and hardware design. The computing world is a combination of different abstraction layers, operating system, networking model, distribution system and graphic libraries are all abstractions, in different levels.

Keep simple and changeable

Simplicity is prerequisite for reliability

– Edsger Dijkstra

Walking on water and developing software from a specification are easy if both are frozen.

– Edward V Berard

Design is the art of arranging code to work today, and be changeable forever.

-– Sandi Metz

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

– C.A.R. Hoare.

The Dutch computer pioneer Dijkstra, had many profound insights on computing complexity and algorithms. He is possibly one of favorite computer scientist in world wide. One of his most cherished habits was in creating amazing articles. Most of the articles written in a fountain pen.

Simplicity doesn’t mean doing less; rather, it’s a way to keep your software maintainable.

When you start writing code, you tend to make it very complex. As you become an experienced programmer, you will find that keeping things simple is the surest way to build complex systems.

Ability to make complex things simple is what sets apart a great programmer from an average one.

Coding

Do things in right way

“Make it work, make it right, make it fast.”

– Kent Beck

When in doubt, use brute force.

——Ken Thompson

The sooner you start to code, the longer the program will take.

—— Roy Carlson

If you can’t write it down in English, you can’t code it.

—— Peter Halpern

Get your data structures correct first, and the rest of the program will write itself.

—— David Jones

Don’t write a new program if one already does moie or less what you want. And if you must write a program, use existing code to do as much of the work as possible.

—— Richard Hill

Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.

– Rick Osborne

We better hurry up and start coding, there are going to be a lot of bugs to fix. 😏

– Anonymous

I’m always happy to follow these principles when programming. These advice helped me to save a lot of time. Remember, do the right thing at proper time.

Optimize it or not

Before optimizing, use a profiler to locate the “hot spots” of the program.

—— Mike Morton

In non-I/O-bound programs, less than four per cent of a program generally accounts for more than half of its running time.

—— Don Knuth

Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

—— Don Knuth

Premature optimization means begin to optimize a program without “hot spot” tracing. You won’t fix performance issue and introduce bugs in this kind of optimization.

Keep code readable

Programs must be written for people to read, and only incidentally for machines to execute.

–Hal Abelson and Gerald Sussman. Structure and Interpretation of Computer Programs

Any fool can write code that a computer can understand. Good programmers write code that humans can understand.

– Martin Fowler

It’s harder to read code than to write it

– Joel Spolsky

Don’t comment bad code. Rewrite it.

– Brian Kernighan

Good code is its own best documentation. As you’re about to add a comment, ask yourself:

‘How can I improve the code so that this comment isn’t needed?’

– Steve McConnell (Code Complete)

If the code and the comments disagree, then both are probably wrong.

—— Norm Schryer

When explaining a command, or language feature, or hardware widget, first describe the problem it is designed to solve.

—— David Martin

Code is like humor. When you have to explain it, it’s bad.

-– Cory House

Do you have trouble in reading the code written by yourself two years ago?

A single piece of code will, over its lifetime, be read hundreds, maybe thousands of times. It will be read hundreds or thousands of times because it must be understood.

Comments will help much on making code readable. Good programmers will write easy to understand the code, and don’t care whether the machine can run, compilers and interpreters will keep your code is right in syntax.

But too much of comments also will not help. If the code is self-explanatory, there is no need for comments. Even if you do need a comment, the comment should be about why you did it, not about what you did.

When writing code, it is better to be clear than to be clever. “Be cleaver” is something like: condensing multiple lines of code into one, using those tricky algorithms, or using some obscure feature of programming language to accomplish a task in a novel way. Tricky code will make it hard to maintain.

Testing

Testing can show the presence of bugs, but not their absence.

—— Edsger W. Dijkstra

If debugging is the process of removing bugs, programming must be the process of putting them in.

– Edsger Dijkstra

Testing leads to failure, and failure leads to understanding.

– Burt Rutan

It takes 3 times the effort to find and fix bugs in system test than when done by the developer. It takes 10 times the effort to find and fix bugs in the field than when done in system test. Therefore insist on unit tests by the developer.

– Larry Bernstein

There is no doubt about the importance of testing. I’m afraid to maintain a code base which don’t contains enough test cases. We should try to find bugs in development phase as many as possible.

Unit testing, integrated testing, fuzzing testing are all good practices to improve the coding quantity. From my experience, the testing code is also document for code, which helpful for others understanding code.

Embrace testing, it will save you much of time.

Deugging

Of all my programming bugs, 80% are syntax errors. Of the remaining 20%, 80% are trivial logical errors. Of the remaining 4%, 80% are pointer errors. And the remaining 0.8% are hard.

—— Marc Donner

The first step in fixing a broken program is getting it to fail repeatably.

—— Tom Duff

Programming is like sex. One mistake and you have to support it for the rest of your life

– Michael Sinz

Debugging is a last-ditch effort to save the code. Debugging code is more difficult than writing code. Because when we need to debug, it means the error have escaped from coding, reviewing, testing.

Usually, find the root cause of a bug is much harder than fix it. If you have reproduced a bug, you almost finished 90% of work.

I’m a fan of Printf Debugging(a.k.a Caveman Debugging).

The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.

-— Brian Kernighan, “Unix for Beginners” (1979)

Debuggers don’t remove bugs. They only show them in slow motion.

– Anonymous

Productivity

Don’t reapeat yourself. Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”

– Andy Hunt and Dave Thomas

I’m not a great programmer; I’m just a good programmer with great habits.

– Kent Beck

Lazy programmers are good programmers, who want to do avoid duplication and won’t do thing reputably.

If there is a lot of repetition in the code, it is likely that we should spend time to refactor the code. Most repeated task is more suitable finished by machine, so we should make it automation.

Tools

The most disastrous thing that you can ever learn is your first programming language.

– Alan Kay

A language that doesn’t affect the way you think about programming is not worth knowing.

― Alan J. Perlis

Programming languages, editors, libraries, all these are tools for programmers.

Learning

The only way to learn a new programming language is by writing programs in it.

– Dennis Ritchie

The first principle is that you must not fool yourself and you are the easiest person to fool.

―- Richard P. Feynman

Avoid “cookbook programming”, where you copy and paste bits of code that you have found to make something work. Make sure you fully understand what everything it actually doing, and that you are comfortable applying the techniques in other situations.

– John

Computer science education cannot make anybody an expert programmer any more than studying brushes and pigment can make somebody an expert painter.

– Eric S. Raymond

Programming isn’t about what you know; it’s about what you can figure out.

– Chris Pine

Theory is when you know something, but it doesn’t work. Practice is when something works, but you don’t know why. Programmers combine theory and practice: Nothing works and they don’t know why.

– Anonymous

When I learned programming, I was also anxious about knowing the details of everything, language syntax, libraries.

We have a ton to lear. This way of learning will make a learner frustrated.

Instead, don’t learn details, learn the essentials and concepts, apply them in practice. Problem-solving is the skill we end up using most.

Finally, don’t lose your curiosity.

Why Copilot Will Not Change Programming

Nic — Sat, 03 Jul 2021 07:35:18 +0000

The hottest tech new of last week is Copilot. Seems almost all programmers are playing or waiting for play with this new fancy AI coding assistant.

Someone says it will totally change the world of programming, an innovative and game-change one!

But, I am not so optimistic about it. From my perspective as an old-school programmer, this kind of tool looks fancy but not so helpful as people give credit for. Not to mention it will be a substitute for real programmers.

Let me explain it to you.

Coding faster is not a big deal

Think about it. How much time do you spend on coding?

In fact, the time we spend on writing code is not a high proportion of the overall software development cycle, even less than a quarter(especially in some tech giants).

We spend much more time on requirements analysis, technical design, testing, debugging, team collaboration.

Software engineering is hard, mostly because of the high complexity of the real world. We can’t change this in truth, abstractions in computing over time are inevitable. And no project ever failed with slow coding.

Coding faster will not make you a better programmer, instead, the ability of abstraction, design, and real experience of real-world engineering will make you a better programmer.

Some aged programmers typing slowly for bad eyesight, but they write bug-free code.

Will help on coding faster?

Years ago I was fascinated by this kind of programming aids. There is a popular template system in Emacs called snippets. We can define our own templates and functions, and then a shortcut would generate code template for me, and I’d just need to fill in the rest.

That looks cool, really cool…

It makes me like a movielike super programmer.

But after I got deep into it, it turns out not as useful as I thought.

tabnine is a similar AI tool with Copilot, which will generate part of the code based on your previous inputs. Copilot will even suggest based on you comments, name of function.

The problem is, when I’m coding with it, I need to constantly review whether the completion is right.

This is a kind of distraction for me. Imagine that, you are an experienced and solid programmer, this AI-tools is a naïve and occasional dumb one, he will pair coding with you.

No, I don’t want it!

The best practice to make programming faster is:

focus, focus, and focus.
Think clearly before coding

Shortcut, variable completion, class attribute, and method names completion are all helpful for coding faster, it will help your typing to catch up with thinking.

But code generation is a different thing. It will distract you, even more than 50% of the completion catches your coding intention, the other nonsense part will slow you down.

If the code base which contains a lot of repetitive pieces or logic, this kind of tool can definitely speed up your coding. Such as most front-end programming, web programming which contains lots of similar CRUD logic.

But I don’t believe this stuff will make a fundamental change to programming.

Will help on learning programming?

Copilot will help to reduce the duplicated tasks like search code from Web. As a beginner to learn programming, it may help you, since when you type some keywords, the code snippet will show up in front of you.

But, as a reminder, copy and make it work is not a better way to improve your coding ability or qualify of work.

Instance coding suggestion may make you happy when coding, but will not help you in long term.

Programming is a craft such as writing, we are writing source code to express ideas. Do you think it’s a good idea to let AI to assist a learner what the next sentence or paragraph will be?

Word completion will help, and sentence auto-completion will become a burden.

Same as writing, if we want to be better at programming, the only way is coding a lot and reading a lot.

John Carmack when talking about learning programming:

Avoid “cookbook programming”, where you copy and paste bits of code that you have found to make something work. Make sure you fully understand what everything it actually doing, and that you are comfortable applying the techniques in other situations.

The potential risks

There are several kinds of risk come with AI coding assistance:

Potential bug introduced in. If the user didn’t pay attention to review the generated code, it will be a nightmare.
The copyright and ownership issue of code. If a code snippet generated from GPL authorized code base and adapted by a commercial company. Whose fault? And who is the owner when the code is generated?
Security. Is it possible for hackers to train some code snippets intensively and induce attacking code to users?

The potential benefits

Of course, this tool may also give some benefits in some scenarios.

To make the tool more useful, we may trend to write more meaningful names and better documents.
Interview will be easier if we allowed to use it?
Make you not longly when programming, it will give you a sense that someone is always with you. Maybe feel better than a small yellow duck.

Top 5 VSCode Extensions for Markdown

Nic — Sat, 29 May 2021 07:26:52 +0000

I moved from Emacs to VSCode for almost one year. Most of the time, I’m editing Markdown.

Let me share these 5 extensions which are useful for boosting your productivity on Markdown.

Markdown All in One

Markdown All in One is a must one for editing Markdown. It contains Markdown preview,

keyboard shortcuts, auto preview, etc.

The shortcuts will make editing Markdown much easy:

The completion also contains pathfinding.

We can print Markdown to HTML with the command: Markdown: Print current document to HTML.

Furthermore, it comes with a beautiful Math support:

Dictionary Completion

Dictionary Completion is another extension to help you input English words quicker. Because I’m not a native English speaker, this also helps me to write words correctly:

Markdown does not support quick suggestions in default, we need to add the below snippet to make hints appear when type in a Markdown file:

"[markdown]": { 
    "editor.quickSuggestions": true
}

Paste Image

I’d love to insert more screenshots to my Markdown files. Paste Image helps you save your images from xclip. This extension supports Mac/Windows/Linux.

PlantUML

PlantUML is an awesome extension if you need to draw diagrams.

In PlantUML, diagrams are defined using a simple and intuitive language. When you editing the source files of the diagrams, you can also preview the result:

PlantUML supports all kinds of diagrams used by developers, such as sequence diagrams, object and class diagrams, etc.

Markdown Emoji

We need more smiles!

Markdown Emoji does one simple thing: add some Emoji faces in your document.

But I found a defect for this extension, my Hexo rendered result does not show Emoji correct in HTML. There is another similar extension Emoji solves this issue. And you can even add Emoji in code:

That all, happy editing Markdown!

5 Fancy Shells You Should Have A Try

Nic — Thu, 27 May 2021 19:55:00 +0000

I love command-line because it makes me more productive.

Most of the time, I live in a Shell in daily work. Shell is simply a command-line interface (CLI) that allows users to interact with the operating system.

Since I have worked with command-line so much of time, I tried many different Shells. I find these 5 Shell worth to be known by more people.

Some of them are already production-ready for daily usage, some are still in rapid development. Maybe you will find your favorite one as your next Shell.

Fish Shell

Fish stands for “the friendly interactive shell”.

It is a smart and user-friendly command line Shell for macOS and Linux family OS.

Many fancy features are included by default, such as syntax highlighting, autosuggest-as-you-type, and fancy tab completions.

The configuration file is located at ~/.config/fish/config.fish. We can add custom functions to it. Fish also provides a web interface to configure the file. Run fish_config , the browser will open port 8000 of the machine, and the user can configure fish on the web page.

Fish is Shell with more than ten years, it’s mature and loved by many programmers. I think the most impressive feature is auto-completion.

If you want to use it in daily work, don’t forget oh-my-fish.

Oil Shell

Oil Shell is a new Unix shell. Shell scripting is a domain-specific language for dealing with concurrent processes, text strings, and the file system, it is hard to learn and a nightmare for maintenance.

| Oil is also aimed at people who know say Python or JavaScript, but purposely avoid shell

Since we have so many existing shell programs, you can not ignore them. Oil helps you gradually migrate away from Shell scripting:

bin/osh runs your existing shell scripts.
bin/oil is a brand new language.

The Oil language is a new dialect which is parsed and evaluated like Python or JavaScript, it’s more concrete than Shell scripting.

The funny thing is, this project is implemented in Python, and then automatically translated to C++ with custom tools.

xonsh

xonsh is another Python implemented Shell.

xonsh‘s syntax is more like Python, actually, all Python code is runnable in xonsh. xonsh is based on Python, and with additional syntax added that makes calling subprocess commands, manipulating the environment, and dealing with the file system easy.

Unlike Oil, you can not reuse existing Bash scripts in xonsh.

Nushell

Nushell is a new Shell, written in Rust. It’s inspired by PowerShell in Windows. Rather than thinking of files and services as raw streams of text, Nushell looks at each input as something with structure. In this way, users can make filters with Pipelines easily.

Nushell has not reached release 1.0, but worth having a try for fun.

Zsh

Zsh is fully compatible with Bash and has extremely rich plug-ins.

Zsh was first released in 1990 and is included in many OS. Actually, Mac OS has already replaced Bash with Zsh as the default shell in Catalina.

If you use Zsh, you won’t miss oh-my-zsh. It’s a community-driven framework for managing your zsh configuration, the repo contains hundreds of powerful plugins and beautiful themes.

Bloom Filter: Concept and Implementation

Nic — Mon, 17 May 2021 23:52:00 +0000

Problem

Think about this scenario, given a list of strings with number of N, where N is large than 1 million.
Let's implement a data structure to check whether a target string is in given strings.

If N is not so huge, a HashDict could be used to build a dictionary. When N is huge, memory is the key point for this challenge.

Different data structures will give you different facility and benefits.
To properly use the power and accessibility of the data structures you need to
know the trade-offs of using one.

In this case, we need a data structure which optimized with space. This is where bloom filter can be used.

The Concept

Bloom Filter was proposed by Burton Howard Bloom in 1970. The related paper is: Space/time Trade-offs in Hash Coding with Allowable Errors.

It's an efficiency in time and space with a very clever design. The cost for space efficiency is accuracy, we called it as a probabilistic data structure:

When the returned result is false, it's 100% correct.
When the returned result is true, it may be wrong.

Why? Let's come up with the details of bloom filter.

The bottom data structure of the bloom filter is a bit array(like BitMap). Initially, when the represented set is empty, all bits are 0:

When we initialize the bit array, we use multiple hash functions to compute multiple indexes according the the strings, and set the bit at index positions to 1.
Why we need multiple hash functions? This is the core idea of bloom filter. Think about HashMap, usually we use one hash function to calculate a hash value, but we also store all the keys in HashMap.
Different keys with same hash function may come out with the same hash value, this is called collision.

In bloom filter, we use multiple hash functions to reduce the possibility of collision.

Even the possibility of collision could be very small, it may happen.

For query API, we do similar hash calculations, but when we find one hash bit is not set to 1, we can safely return false. For naive Bloom Filter, the keys added into table can not be deleted. If we want to delete key, a variant named Counting Bloom Filter(CBF) could be used.

We can have some trivial tunning about the data structures, you can modify the false positive rate of your filter. From above description, a larger filter will have less false positives, and more hash functions will introduce more cost in computation. Given a Bloom filter with m bits and k hashing functions with n insertion.

Your false positive rate will be approximately $$ (1-e^{-kn/m})^k $$.

Implementation of C

#include <stdlib.h>
#include <limits.h>
#include <stdarg.h>

typedef unsigned int (*hashfunc_t)(const char *);
typedef struct
{
    size_t asize;
    unsigned char *a;
    size_t nfuncs;
    hashfunc_t *funcs;
} BloomFilter;

#define SETBIT(a, n) (a[n / CHAR_BIT] |= (1 << (n % CHAR_BIT)))
#define GETBIT(a, n) (a[n / CHAR_BIT] & (1 << (n % CHAR_BIT)))

BloomFilter *bloom_create(size_t size, size_t nfuncs, ...)
{
    BloomFilter *bloom;
    va_list l;
    int n;

    bloom = malloc(sizeof(BloomFilter));
    bloom->a = calloc((size + CHAR_BIT - 1) / CHAR_BIT, sizeof(char));
    bloom->funcs = (hashfunc_t *)malloc(nfuncs * sizeof(hashfunc_t));
    va_start(l, nfuncs);
    for (n = 0; n < nfuncs; ++n)
        bloom->funcs[n] = va_arg(l, hashfunc_t);
    va_end(l);

    bloom->nfuncs = nfuncs;
    bloom->asize = size;

    return bloom;
}

int bloom_add(BloomFilter *bloom, const char *s)
{
    size_t n;

    for (n = 0; n < bloom->nfuncs; ++n)
        SETBIT(bloom->a, bloom->funcs[n](s) % bloom->asize);

    return 0;
}

int bloom_check(BloomFilter *bloom, const char *s)
{
    size_t n;

    for (n = 0; n < bloom->nfuncs; ++n)
    {
        if (!(GETBIT(bloom->a, bloom->funcs[n](s) % bloom->asize)))
            return 0;
    }

    return 1;
}

unsigned int sax_hash(const char *key)
{
    unsigned int h = 0;

    while (*key)
        h ^= (h << 5) + (h >> 2) + (unsigned char)*key++;

    return h;
}

unsigned int sdbm_hash(const char *key)
{
    unsigned int h = 0;
    while (*key)
        h = (unsigned char)*key++ + (h << 6) + (h << 16) - h;
    return h;
}

int main() {     
    BloomFiler* bloom = bloom_create(2500000, 2, sax_hash, sdbm_hash);
    ...
}

Implementation of Python

Please refer to Pure Python Bloom Filter module. The usage is straight forward:

from bloom_filter import BloomFilter

bloom = BloomFilter(max_elements=10000, error_rate=0.1)

assert "test-key" in bloom is False

bloom.add("test-key")
assert "test-key" in bloom is True

Real cases

b.com/google/leveldb/blob/master/util/bloom.cc)

5 Tips For Techical Interview

Nic — Sun, 16 May 2021 23:52:00 +0000

As a tech lead worked at several technical giants, I have participated in dozens of interviews as a interviewer. Here I want to share with you some tips in a programming interview.

Be confident and honest

Many candidates (especially for some new-grads), because they don't have much experience on interviewing, they may tend to be unconfident. They cannot express their opinions fully and freely. And when the interviewer challenge them, they will immediately turned to be timid. This is a red flag for interviewer. Because this will give a impression that they are not assertive enough, or their foundation of computer science is not solid.

Another important point is be honest, most interviewer don't want a un-honest colleague. I found a candidate tried to pretend that she knew something on a topic which actually she was not familiar with. She told me a lot stuff but it's wrong, and further more she tried to change the subject since she didn't know the right answer. This seems a clever trick in daily communication, but it is not a good way in a interview. If you don't know the answers for some questions, just admit you don't know it. It's OK, nobody knows all technical details or answer every questions.

Communication is important

Interview is not all about a game of answering questions or examination. Instead, interview is a mocking scenario in daily work. The interviewer is trying to find a human but not a machine. Because in daily work, collaboration is an key part of software development. Although the your answering and coding is important, effective communication is also necessary.

A smooth self-introduction is necessary, practice it until you feel comfortable.

When you faced with challenge that you can't find the solution, don't let the entire interview fall into silence. Instead, tell the interviewers what you know, and what you think, actively ask the interviewer for hints.

If you don’t give out your ideas, we don't know the way of your thinking and the style of your reasoning.

Good questions from you can also show your thinking advantages. I have given HIRE result to some candidates who did not come up with the final answers, because their thinking process are good.

The interviewers prefer to candidate with a clear thinking. We doesn’t just want you to write down the answer, we value the process of how you get there. People who can explain something clearly tend to be a better collaborator.

Write clean and readable code

Write clean and readable code is not easy, this may need some deliberate training, try to learning from great source code.

After finishing your code, you should spend a some amount of time to optimize your code structure to make it more readable.

For instance, it can be better abstracted with appropriate helper functions for duplicated code.

Variable naming is a key point, even the code is usually less than 100 lines in a interviewing.

Appropriate comments will also impress interviewers. But, remember you need to avoid overly abstraction, which will make it very difficult to read.

Add testing for your code

In daily work, add testing is your necessary task for any changes, especially for a complicated system.

Take the initiative to propose corner cases, which reflects your careful thinking and is very important for the positions of software development!

After coding, you should analyze the time and space complexity, and then run some basic examples to verify your solution. If you have time, please add more testing for your code.

Show your passion and curiosity

Many candidates seems treat interviewing as an examination. We prefer to the candidate who has a passion and curiosity in computer science, because these two qualities will make everything easier in their career. Those who truly love programming will find more fun and have a better performance in work. For those who don't love the craft of programming, even if they manage to get a job, they will hate it.

A perfect way to show your passions is your hobby project, or your contributions to the open source community.

Top 5 Books for Python Programming

Nic — Tue, 17 Nov 2020 10:38:37 +0000

Original Post

Python is a language I like very much, and it is also hot right now. In this post, I will list some Python books for beginners and experienced programmers who are interested in Python.

Let’s start from beginner to expert levels.

Python Crash Course

This book is a good introduction to Python. It will teach you the basic syntax, such as variables, lists, classes, control flows, etc. All the chapter contains some handy exercises for readers. In the second half part, we will practice Python with several small projects, play with some handy libraries.

It’s well written and easy to understand. If you are a fresh new sailor, start your Python journey with it!

Fluent Python: Clear, Concise, and Effective Programming

This is the strongly recommended one for those who have got some Python programming experiences.

Fluent Python skips over the basics, instead, this book will focus on those specific features which rarely presented in other programming languages, it also provides enough details about implementations. It felt like a treasure box to me.

I spent about one month to read it. I read it slowly, a few pages every night. It’s a pleasure.

Never has a Python book gave me such an interesting experience. It’s so detailed and informative, the author is really good at writing and teaching.

If you looking to upgrade your Python skills, this is THE book. I promise you no regrets for the time you spend on this book.

Python Essential Reference

This book explains in detail the most important parts of the core Python language and the Python library. It covers types and objects, operators and expressions, programming structures and control flows, input and output, testing, debugging, and more, as well as some advanced topics not covered in the official Python documentation or other references.

Although this kind of book looks a bit boring, it’s definitely beneficial for a Python programmer to read. Or, at least, go through the topics interested you and treat it as a reference book.

The Hacker’s Guide To Python

This book contains more advanced topics. The horizon is no longer on the language details. The author is a Python expert and had led some super large Python projects. There are many thoughts on software engineering, which will benefit to all software engineers, not just Python programmers.

High-Performance Python: Practical Performant Programming for Humans

Python is a programming language with a high level of abstraction, and one of its great strengths is the efficiency of the development, with the cost of default performance.

This book is for those who have already started with Python and want to tune the performance of Python code. It’s mainly focused on performance optimization related topics.

A major goal of this book is to introduce packages and principles that will enable rapid Python development while avoiding many performance limitations, reducing development and maintenance costs, and regain the benefits of system efficiency.

Config Nginx For Security

Nic — Mon, 16 Nov 2020 22:52:00 +0000

Orignal Post

When I was in the security group, we found many programmers don't know how to handle, or don't have the awareness of Nginx's security. We found many exploitations or cases of sensitive data was pulled from web servers by attackers.

Here I will list some configurations for Nginx's Security. Please do adjustments to these configurations according to your actual requirements.

Forbidden any sensitive request path

Many source codes are using Git as version control tools. The directory .git will contains all the metadata of codebase, including some secrets, etc. Expose it to the public will make your codebase downable for attackers:

So we need to disable the .git related request routes.

location ~ /\.git {
  deny all;
}
## Disable .htaccess and other hidden files
location ~ /\.(?!well-known).* {
    deny all;
    access_log off;
    log_not_found off;
}

Forbidden unnecessary HTTP request methods

The most commonly used HTTP request methods are GET, POST, HEAD. We should return 444 for any other unused methods.

## Only allow these request methods ##
if ($request_method !~ ^(GET|HEAD|POST)$ ) {
         return 444;
}
## Do not accept DELETE, SEARCH, and other methods ##

Add request rate limiting

Rate limiting will block many malicious requests, and it is also a frequently used tool to defend against network and application-level DDoS attacks against websites.

We can add the maximum request limit for a single IP.

limit_req_zone $binary_remote_addr zone=ip:10m rate=5r/s;

server {
    listen 80;
    location / {
        limit_req zone=ip burst=12 delay=8;
        proxy_pass http://website;
    }
}

Add HTTPS for your service

If you are deploying a non-company project, Let's encrypt will be enough for personal usage. And remember to redirect all non-HTTPS requests to HTTPS with 301.

server {
    listen               80;
    listen               443 ssl;
    server_name          www.coderscat.com;
    ssl_certificate      /etc/letsencrypt/live/coderscat.com/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/coderscat.com/privkey.pem;
    return 301 https://coderscat.com$request_uri;
}

server {
    listen 80;
    listen 443 ssl http2;
    server_name coderscat.com;

    # ssl on;
    ssl_certificate       /etc/letsencrypt/live/coderscat.com/fullchain.pem;
    ssl_certificate_key  /etc/letsencrypt/live/coderscat.com/privkey.pem;

    ssl_protocols        TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers          ECDHE-RSA-AES128-GCM-SHA256:HIGH:!aNULL:!MD5:!RC4:!DHE;
    ssl_prefer_server_ciphers  on;
    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout  10m;
}

Enable SELinux

SELiunux stands for Security-Enhanced Linux, it is a Linux kernel security module that provides a mechanism for supporting access control security policies, including mandatory access controls (MAC). It can prevent several attacks before your system being hacked.

Install it on Ubuntu:

$ sudo apt install policycoreutils selinux-utils selinux-basics
$ sudo selinux-activate
$ sudo selinux-config-enforcing  (Then reboot system)
$ sudo sestatus (query the status of SELinux)

Clickjacking Attack

Clickjacking attack will cause users to unwittingly download malware, visit malicious web pages, provide credentials or sensitive information.

We can inject X-FRAME-OPTIONS in HTTP Header to prevent a clickjacking attack(Even this can be bypassed in some ways). This is achieved by adding below in nginx.conf file

add_header X-Frame-Options "SAMEORIGIN";

Above header will instruct a browser to load the resources ONLY from the same origin.

X-XSS Protection

Inject HTTP Header with X-XSS protection to mitigate Cross-Site scripting attack. Modify nginx.conf file to add the following

add_header X-XSS-Protection "1; mode=block";

Save the configuration file and restart Nginx. You can use the Headers Test tool to verify after implementation.

You may also be interested in implementing OWASP recommended secure headers which are explained here.

References

Mastering NGINX Second Edition is an excellent reference book for the introduction of Nginx.