Whenever I see a flock of birds roaming in the sky, I will stare and admire them for several seconds. My favorite thing to do while scuba diving is to swim along or stay inside a huge school of fishes and watch them swim around me. There is something mesmerizing about the movements, not only as an individual bird but also as an entire group. 

When I heard about a simple algorithm that only uses 3 factors to simulate the movements of flocks, it had my interests and I started writing an implementation of it.

It is [live here](https://luuu-xu.github.io/boids-canvas/) and as an iframe element below, go ahead and play around with it. 

If you are interested in the code, its GitHub repo is [here](https://github.com/luuu-xu/boids-canvas).

<iframe
  title="Live Example of Boids on Canvas"
  width="100%"
  height="500px"
  loading="lazy"
  src="https://luuu-xu.github.io/boids-canvas/">
</iframe>

## Boids Algorithm

The [Boids Algorithm](https://en.wikipedia.org/wiki/Boids) developed by Craig Reynolds in 1986 is a smartly designed flocks simulating model that only uses 3 rules. In my implementation, I followed these 3 elegant rules and coded with OOD in mind to better capture the individual boid's movements.

### 1. Alignment

The steer force that each boid takes to align towards the average direction of the local flocks. 

In my implementation, the average velocity within local flocks is calculated and normalized, then added to boid's new velocity.

### 2. Cohesion

The steer force that moves boid to the average position of the local flocks. 

This rule is similarly implemented by calculating average position (center of mass) of local flocks, then the normalized velocity required to reach that average position is added.

### 3. Separation

The steer force that each boid uses to avoid the local flocks. 

The implementation is that the by setting a parameter called separationRadius, we fetch all boids within the radius, and add a separation force avoiding it, summed up to a new velocity.

## Simulating a Bird

Before putting Boid's Algorithm onto the boid class, I started with creating a bird-like class that can be animated using HTML [Canvas API](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API).

Firstly, the boids would move towards a random direction because of an initial velocity and position was given to them. The Canvas API's [requestAnimationFrame()](https://developer.mozilla.org/en-US/docs/Web/API/window/requestAnimationFrame) function was called recursively so that the animation kept going, the initial velocity was added onto the position every frame, animating the boids.

Secondly, I thought the boids/birds should not move in a straight line, but with some random changes in direction. So a changeDirectionRandomly() method is added to the Boid class. It needed a timer and a parameter which gave the boid a new velocity such the new direction is adjusted with the parameter. 

Then, the boids needed to move away from the walls in a realistic way. The solution was that a steer force would be gradually added to the velocity as the boid approaches the walls, i.e. the closer to the wall, the stronger the steer force. This was actually quite an easy solution to avoiding walls, the end result was nice enough.

After these steps, I wrote the three rules of Boids Algorithm: alignment, cohesion, separation to the Boid class and added these new combined velocity before moving the boid. The boids started to act like flocks. With the ability to avoid from walls, the flocks looks realistic enough and the simulation result was great.

## Control Panel

In order to grasp the rules of and understand the algorithm better. I decided to add a control panel to the page such that several critical parameters could be adjusted in real time where users would play around with the simulation and feel the impacts of different parameters.

These five adjustable parameters were added in the control panel with appropriate UI:

1. Number of Boids
2. Maximum Speed
3. Radom Direction Changing Factor
4. Perception Radius
5. Separation Radius

The parameters changes were in real time. The slider component needed a debounce functionality for better rendering and performance, thus I added a 500ms debounce to them. It was really fun to watch how flocks started to gather around into bigger crowd, split to smaller groups, change to different speeds as we adjusted the factors. 

## Throttle to 60 FPS

All of these code and methods are working great. But when I was playing around with the control panel, I found out the strange behavior of boids having a higher speed every time I adjusted a parameter despite of my speedLimit() method on Boid class.

After some debugging and readings into Canvas API, I figured the culprit was a higher FPS every time a parameter was adjusted. Then I added a real time FPS display to the page and it proved me correct. The canvas was rendering at a higher FPS gradually and since the visual speed of boids are FPS times velocity, they looked like they are moving faster and faster.

The solution was quite simple, a throttle of 60 FPS is set for the animation. It implemented such that with a time gap that is lower than 1000 / 60 = 16.66ms, the animation would render the next frame, giving us the throttle we needed.

After the fix, boids would move at constant visual speed regardless of parameter adjustments, but only responded to the adjustment of maximum speed.

## Some Comments

It is quite interesting that three simple rules applying to individuals can generalize the movements for groups also. 

While looking at flocks of birds or schools of fishes, the balance between individuality and similarity is easily relatable to the observer whereas such an individual person lives under social norms. 

The three rules of Boids Algorithm: alignment, cohesion and separation could also be applied to ourselves examining how we interacted with surrounding crowds and made changes to better adapt into the group.


![Book Cover](/blogImages/designing-data-intensive-applications-1/book-cover.jpg)

To be honest, I do not enjoy taking down reading notes, except occasionally highlighting on my Kindle for some great paragraphs. But when I opened [Designing Data-Intensive Applications](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/) and saw the estimated reading time being **25 hours**, I said to myself, "Maybe I should make reading notes along the way to make these 25 hours really worth it." 

So this blog is going to be the place where I summarize this book by chapters, the points will be what strikes me as "worth highlighting", thus it won't be a full summarization of this book at all.

***

## Chapter 1. Reliable, Scalable, and Maintainable Applications

Three most important concerns in most software systems:

### Reliability

- Reliability can mean roughly, "continuing to work correctly, even when things go wrong."

- Kinds of faults that can be cured: Hardware Faults, Software Faults, Human Errors.

### Scalability

#### Load

- Load is described with **load parameters**. These depend on the architecture of the system: it can be requests per second to server, the ratio to writes in a database, the number of simultaneously active users, etc.

- Twitter's first version used the approach where new tweets are inserted into global collections, user requests their home timeline, the tweets are looked up and merged together. This is fast on posting but slow on reading.

- Twitter's second version used another approach where a new tweet is sent to a cache for each user's home timeline "mailbox". This approach is going to be slow on posting but super fast on reading. But this approach makes sense because the rate of posting tweets is **two orders of magnitude lower** than fetching home timeline reads.

- But celebrities have a huge number of followers where second version will have a huge slow down, thus Twitter uses a hybrid approach where celebrities' tweets are fetched separately like version 1 and merged into users' home timeline mailbox in version 2.

#### Describing Performance

- **Response time** (time between a client sending a request and receiving a response) is more important than **throughput** (the number of records that can be processed per second) in online systems.

- High percentiles of response times, **tail latencies**, are important for Amazon because these customers are the ones with the most number of order histories which indicates that they are the most valuable customers.

### Maintainability

- Three aspects of maintainability: Operability, Simplicity ([Clean Code](/blog/clean-code)), Evolvability.

***

## Chapter 2. Data Models and Query Languages

### Relational Model Versus Document Model

- Relational databases' roots are within **business data processing**. The use cases were either **transaction processing** and **batch processing**.

- Example of a LinkedIn resume data. The **one-to-many relationships** from the user profile to the user's positions, educational history, etc, imply a tree structure in the data, and [JSON](https://en.wikipedia.org/wiki/JSON) representation makes this structure explicit.

- Document databases favor in schema flexibility, better performance due to locality, and to some applications it is closer to the data structure.

- Relational databases counter by providing better support for joins, and many-to-one and many-to-many relationships.

### Query Languages for Data

- **Declarative query** language like SQL, you specify the pattern of the data you want, but not *how* to achieve the goal. The database system's query optimizer decides which indexes and which join methods to use and the order to execute parts of query.

- **MapReduce** is implemented in MongoDB by [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline/) where it uses a JSON-based syntax rather than SQL's sentence-style syntax. The difference is maybe a matter of taste.

### Graph-Like Data Models

- A graph consists of **vertices** (nodes) and **edges** (arcs). Examples include social relationships graph, web pages graph and rail networks graph.

- Many algorithms focus on graph, check out some of them on [LeetCode](https://leetcode.com/tag/graph/).

***

## Chapter 3. Storage and Retrieval

### Data Structures That Power Your Database

- An important trade-off in storage system: well-chosen indexes speed up read queries but slows down writes. So normally database administrator chooses indexes manually using knowledge of the application's typical query patterns.

#### Hash Indexes

- Indexing strategy: keep an in-memory hash map where every key is mapped to a byte offset in the data file, which is the location at which the value can be found. When you append a new key-value pair to the file, you also update the hash map to reflect the offset of the data just wrote.

- When deleting a key and its value, you have to append a special deletion record to the data file (tombstone). When the segments are merged, the tombstone tells the process to discard this key-value pair.

#### SSTables

- **Sorted String Table** (SSTable), requires that the sequence of key-value pairs is *sorted* by *key*. Additionally it requires that each key only appears once within each merged segment file.

- Now with SSTables, merging segments is simple and efficient just like [mergesort algorithm](https://en.wikipedia.org/wiki/Merge_sort).

- Find a particular key is now easier because keys are sorted so you can look for a key in between two closest keys.

- Constructing and maintaining a sorted structure can be done with [red-black trees](https://en.wikipedia.org/wiki/Red%E2%80%93black_tree) or [AVL trees](https://en.wikipedia.org/wiki/AVL_tree).

- When a write comes in, add it to in-memory balanced tree data structure, this is called a **memtable**.

- When a memtable gets bigger than a threshold, write it out to disk as an SSTable file. This new SSTable file becomes the most recent segment of the database.

- When a read request comes in, first try to find the key in the memtable, then in the most recent on-disk segment, then the next-older segment, etc.

#### LSM-Trees

- **Log-Structured Merge-Tree** (LSM-Trees), builds on log-structured filesystems. Storage engines that are based on this principle of merging and compacting sorted files are called LSM storage engines.

- LSM-tree algorithm can be slow when looking up keys that do not exist in the database: you need to check the memtable, then the segments all the way back to the oldest. 

- To optimize this access, storage engines often use [Bloom Filters](https://en.wikipedia.org/wiki/Bloom_filter), which is a memory-efficient data structure for approximating the contents of a set. It tells you if a key does not appear in the database, saving unnecessary disk reads.

#### B-Trees

- B-trees break the database down into fixed-size **blocks** or **pages**, and read or write one page at a time. It is different comparing to previous indexes which has variable-size segments.

- One page is considered the **root** of the B-tree. Whenever looking for a key, you start from the root. This root page contains several keys and references to child pages. Each child is responsible for a continuous range of keys. 

- When looking for a key, starting from root, eventually we will reach a **leaf page**, which contains the references to the pages where values can be found.

- When adding a new key, you need to find the page whose range encompasses the new key and add it to that page. If there isn't enough free space in the page for the new key, then this page is split into two half-full pages, and the parent page is updated to account for the new subdivision of key ranges.

- This algorithm ensures the tree remains **balanced**, and a B-tree with n keys always has a depth of O(log n).

- Write operations of B-tree is to overwrite a page on disk, while log-structured indexes such as LSM-trees only appends to files but never modify files in place.

- In order to make B-tree database resilient to crashes, it is common to include an additional data structure: a **write-ahead log** (WAL). This file is append-only, which needs to be appended before every B-tree modification. When the database comes back after a crash, this WAL is used to restore the B-tree back to a consistent state.

#### Comparing B-Trees and LSM-Trees

- One write to the database resulting in multiple writes to the disk is known as [write amplification](https://en.wikipedia.org/wiki/Write_amplification). LSM-tree indexes rewrite data multiple times due to repeated compaction and merging SSTables; while B-tree indexes must write every piece of data at least twice, once to the WAL, and once to the tree page itself.

- LSM-trees can be compressed better, and produce smaller files on disk than B-trees.

- B-tree storage engines leave some disk space unused due to fragmentation, when the page is split, some space in a page remains unused.

- LSM-trees has a downside where the compaction process can sometimes interfere with the performance of ongoing reads and writes.

- An advantage of B-trees is that each key exists in exactly one place in the index, whereas a LSM-tree may have multiple copies of the same key in different segments.

#### Other Indexing Structures

- When the value in an index is a reference to the row stored elsewhere, the place where rows are stored is known as **heap file**, and it stores data in an no particular order. This approach is common because it avoids duplicating data when multiple secondary indexes are present.

- In some situations, the extra hop from the index to the heap file is too much of a performance penalty for reads, it is desirable to store the indexed row directly within an index, which is known as **clustered index**.

- **Multi-column index** is also called concatenated index, which simply combines several fields into one key by appending one column to another.

- **Lucene**, the in-memory index is a finite state automation over the characters in the keys, similar to a [trie](https://leetcode.com/problems/implement-trie-prefix-tree/). This automation supports efficient search for words within a given edit distance.

- Because many databases are not that big, it is feasible to keep them entirely in memory, which is **in-memory databases**. These in-memory databases write and read from memory, while writing an append-only log for durability to disk too.

### Transaction Processing or Analysis Processing?

- A transaction needn't necessarily have [ACID](https://en.wikipedia.org/wiki/ACID) (atomicity, consistency, isolation, durability) properties. **Transaction processing** just means allowing clients to make low-latency reads and writes - as opposed to **batch processing** jobs, which only run periodically.

- **OLTP** (online transaction processing) typically looks up a small number of records by some key with an index. Records are inserted or updated based on user's input.

- **OLAP** (online analytic processing) usually scans over a huge number of records, only reading a few columns per record, and calculate and aggregate statistics rather than returning raw data. They are often written by business analysts to gain business intelligence (BI).

- A [data warehouse](https://en.wikipedia.org/wiki/Data_warehouse) is a separate database that analysts can query without affecting OLTP operations. The data warehouse contains a read-only copy of the data in all the various OLTP systems in the company. Data is extracted from OLTP databases (periodic data dump or continuous stream of updates), transformed into an analysis-friendly schema, cleaned up, and then loaded into the data warehouse. This is known as [Extract-Transform-Load (ETL)](https://en.wikipedia.org/wiki/Extract,_transform,_load).

- Many data warehouses are used in a style known as [star schema](https://en.wikipedia.org/wiki/Star_schema) (dimensional modeling). At the centre of the schema is a **fact table**. Each row of the fact table represents an event that occurred at a particular time.

- Some columns in the fact table are attributes, like price of a product. Other columns in the fact table are foreign key references to other tables called **dimension tables**.

- A variation of star schema is called [snowflake schema](https://en.wikipedia.org/wiki/Snowflake_schema), where dimensions are further broken down into sub-dimensions.

### Column-Oriented Storage

- The idea behind **column-oriented storage** is simple: don't store all the values form one row together, but store all the values from each column together instead. This way, querying only needs to read and parse columns that are used in that query, which saves a lot of work.

- Column-oriented storage leads very well to compression. One technique that is particularly effective in data warehouses is [bitmap encoding](https://en.wikipedia.org/wiki/Bitmap_index) which makes encoding of a column remarkably compact.

- Administrators can choose to sort column storage as an indexing mechanism. For example, if queries often target data ranges, it makes sense to sort the entire column storage with date as the first key. This will make query optimizer scanning only the rows needed.

- A second sort key can be imposed as well for the values with the same first key, for example, after sorting with date, we can sort secondarily with product.

- Writing to column storage usually uses [LSM-trees](#lsm-trees). All writes first go to an in-memory store, where they are added to a sorted structure and prepared for writing to disk.

***

## Chapter 4. Encoding and Evolution

### Formats for Encoding Data

- The translation from in-memory representation to a byte sequence is called **encoding** (**serialization**), and the reverse is **decoding** (**parsing**, **deserialization**).

#### JSON, XML, and Binary Variants

- In XML and CSV, you cannot distinguish between a number and a string that consists of digits. JSON distinguishes strings and numbers, but it doesn't distinguish integers and floating-point numbers.

- Binary encoding is suitable to use internally within organization, one of the implementation is using Protocol Buffers like [gRPC](https://grpc.io/about/).

- JSON is less verbose than XML, but both use a lot more space compared to binary formats. This leads developments of profusion of binary encoding like [BSON](https://www.mongodb.com/basics/bson#:~:text=BSON%20stands%20for%20Binary%20Javascript,binary%20formats%2C%20like%20Protocol%20Buffers.) for JSON, [WBXML](https://www.w3.org/1999/06/NOTE-wbxml-19990624/) for XML.

#### Thrift and Protocol Buffers

- Thrift and Protocol Buffers are binary encoding libraries that require a schema for any data that is encoded.

#### Avro

- Avro is another binary encoding format with two schemas: one (Avro IDL) for human editing, one (based on JSON) more easily machine-readable.

- It has **writer's schema** when encoding, and decode the data with **reader's schema**. Key idea of Avro is that writer's schema and reader's schema **don't have to be the same** but only to be compatible.

### Models of Dataflow

#### Dataflow Through Databases

- In a database, the process that encodes the data also decodes it, just like *sending a message to your future self*. Thus **backward compatibility** is necessary.

- A value in the database may be written by a newer version of the code, and read by an older version of the code that is still running. Thus **forward compatibility** is often required too.

- *Data outlives code* happens when old data lives in the original encoding and never rewritten. Rewriting (migrating) is possible but expensive, most databases avoid it.

#### Dataflow Through Services: REST and RPC

- Most common arrangement is to have: **clients** and **servers**. The servers expose APIs over network, and the clients connect to the servers with requests to the APIs.

- A server can itself be a client to another service, which is called *microservices* architecture.

- **Web Service** is when HTTP is used as the underlying protocol. Two popular approaches are **REST** and **SOAP**.

- REST emphasizes simple data formats, using URLs for identifying resources and using HTTP features for cache control, authentication, and content type negotiation.

- SOAP is XML-based protocol that aims to be independent from HTTP and avoids using most HTTP features. The API of a SOAP web service is described using [WSDL](https://www.w3schools.com/xml/xml_wsdl.asp) (Web Services Description Language).

- **RPC** (Remote Procedure Call) tries to make a request to a remote network service look the same as calling a function or method within the process.

- But RPC is fundamentally flawed because network request is very different from a local function call:

  - Network request is unpredictable, the request or response may be lost. 

  - A local function call either returns a result, or throws an exception, or never returns. But network request may return without a result, due to a *timeout*.

  - When you retry a network request, it maybe that the request is getting through but only the responses are getting lost.

#### Message-Passing Dataflow

- **Message Queue** is similar to RPC where a client's request is delivered to another process with low latency and similar to databases where the message is sent via an intermediary called a **message broker**.

- It is usually one-way, sender doesn't expect to receive a reply to its messages. This communication pattern is **asynchronous**, the sender sends the message and forgets about it.

- Popular message brokers are [RabbitMQ](https://rabbitmq.com), [Kafka](https://kafka.apache.org), etc.

- Normal usage: one process sends a message to a named **queue** or **topic**, and the broker ensures the message is delivered to one or more **consumers** or **subscribers** to the queue or topic.

***

For this first part of the book, our main focus is data stored in a single machine, the next part will be storing them in multiple machines:

Continue to the second part of the book: [Distributed Data](designing-data-intensive-applications-2).


![Wtfs per minute](/blogImages/clean-code/wtfs-per-minute.png)

Writing code is more of a dedicating craft than a mythically creative process. Being a craft, just like leather-working or carpentry, programming skills need to be seriously regarded, constantly improving, ever stopping adapting and learning. Skills can be learnt and honed, mindsets can also be learnt though through a more difficult path, but attitude towards this craft is crucial and will be shown from details and corners of the works and products.

## Hand-making Leather Products

For the past few years I have been hand-making custom ordered leather products for clients in the world, from bi-fold wallets to women's handbags. When I am talking about my leather-working skills and practices, I am indeed regarding them as results and showcases of my dedication and craftsmanship. 

In order to attract clients, I need to differentiate my works from 99.9% of the leather products in the market which are mass-produced, highly industrialized, and with little care and attention. Clients who seek this craftsmanship will look for makers themselves, by examining the products, communicating with the makers and eventually placing an order to feel the product in their own hands.

I pay all of my attention while hand-making a custom order for my client, with my best efforts by far reaching a higher standard, simply because I take pride in what I do, and I am being extremely considerate to my client. Being considerate to the client means that I am making a product with the standard that I would be happy with if I were the client. I wouldn't want to be upset or unsatisfied about the custom order that I had been waiting for two months myself.

Transferring from leather-working to coding, being considerate is still required. Although the person that I am imaging myself in differs now, from the client who waits for my product, to the next person who reads and maintains the code I just wrote.

## Hand-writing Codes

Because code base is most likely ever evolving, with additions of features, removals of bugs, adaptations of newer frameworks and optimizations from better tools. I would be better off if I am considerate of the next person who reads and adds into this code base, imaging myself to be this next person. Yes, for personal projects I have built, the next person is possibly me. But for a collaborative project, I would be the "next person" in the first place when I initially started looking into the code, and I would be responsible for making the next person easier to understand what I have changed.

Reading [Clean Code: A Handbook of Agile Software Craftsmanship](https://www.oreilly.com/library/view/clean-code-a/9780136083238/) was easy and fun, the concepts align with what I agree and strive for and most importantly the techniques and pointers are informative and helpful. There are few techniques that stick with me strongly, I would like to list them below:

### 1. The boy scout rule

"Leave the campground cleaner than you found it." This rule is easy to understand, but it is merely the general goal we are trying to achieve every time. It is also a mindset that ultimately become mutual-beneficial to ourselves and others. There are many techniques and principles that will help us to adhere to this rule.

### 2. Single Responsibility Principle (SRP)

Robert C. Martin expresses SRP as "a class or modules should have one, and only one, reason to change." This principle ensures that a module or function should focus on one thing, in order to minimize the scope of it. This will bring in quite some best practices that are easy to understand and remember too:

- Do one thing: try refactoring complex functions to small ones which ultimately does one thing only.
- Write small functions, modules or class.
- Give meaningful and descriptive names to functions and variables.
- Try passing as few as possible arguments, 1 is better than 2, 2 is better than 3.
- Don't use flag argument, you can easily split this function into two.

### 3. Express yourself in code instead of comments

In my own codes, from front-end React hook logics to algorithm steps, I have done it: using comments rather than code to explain myself. This is suboptimal, a mistake for sure and it is easy to understand too. Only when our code is not explanatory enough, we will try using long comments to supplement the code.

Refrain from writing comments, try giving more explanatory names to variables and functions, and making sure a function only has one level of abstraction where logics are easier to understand. 

Some comments are good, like a general introduction to an algorithm explaining why it works. Then make sure the actual code does the rest explaining the details and implementations.

### 4. Don't Repeat Yourself (DRY)

Codes should be efficient, repeated codes are not. When we see similar or repeated functions, we should try rewriting them so we can avoid the unnecessary repetition. We can modularize the functions, where it does only one thing, but it can be implemented to different places and abstractions as needed. Because this module does one thing, it is easy to change and maintain, ready for testing and provides better documentations.

### 5. Test Driven Design (TDD)

I am not sure whether TDD or Clean Code is the reason for the other. Writing codes with TDD in mind is crucial for maintainability. If we have tests covering for every functions we have, we are no longer invulnerable towards introducing bugs or breaking anything. There are again a few best practices we can follow:

- Write test first, then code, don't write more code until a failing test is present.
- Tests need to be small, simple and readable.
- Make sure tests are guaranteed to be repeatable.
- Use a testing framework that is fast, optimally using one command line to run all of them, this ensures us that we are incline to actually run these tests.

## Your Product Shows Your Craftsmanship

There is a saying about great cabinet makers, "Look at the back of the drawers, that is where you know it's worthy." This saying also sheds some lights and guides me into both coding and leather-working. You look at the code bases and the insides of my leather products as if they were the back of the drawers.

For both of them, coding and leather-working, I intend to be proud of what I am offering to others. This proud is woven into the products I am building, users and clients will look closely and feel this craftsmanship.

> There are two parts to learning craftsmanship: knowledge and work. You must gain the knowledge of principles, patterns, practices, and heuristics that a craftsman knows, and you must also grind that knowledge into your fingers, eyes, and gut by working hard and practicing.

I have learned basic and advanced techniques in leather-working, acquired my craftsmanship by perfecting practices and provided great products to many clients. And I am now on the journey of acquiring another craft, the making of a software or web app, again I intend to do so by learning the knowledge and putting them to use in hard-works.

---

This blog post is also published on Medium [here](https://luuu-xu.medium.com/clean-code-and-leather-working-1097792d6a90). 

[Clean Code and Leather-working](https://luuu-xu.medium.com/clean-code-and-leather-working-1097792d6a90)



![A typewriter](/blogImages/markdown-blog/laura-chouette-6tdfx_gvHf0-unsplash.jpg)

In this blog post I am going to show you how I implemented this [Markdown blog](/blog) feature on my Next.js portfolio website. This is going to be my first "tutorial-like" blog, hopefully this will make sense to anybody reading it and I am going to try my best.

I am going to skip the introduction why you need a markdown blog feature on your website (since you are reading this), I am assuming you want it. And I am going to show you a list of what this markdown blog feature does and why it is worthy recommending.

## Features

1. A [BlogIndex](/blog) page where a list of blogs are displayed with Title, Published Date and more.
2. A single page for each [Blog](/blog/markdown-blog) where the `<article />` has the Markdown content and is styled automatically.
3. Write your blog in `example.md` markdown file and store any **images/screenshots** alongside and the lib function will copy them into the `/public` folder for Next.js to render.

## File Structure

Assuming you already have your portfolio website built with Next.js, the structure relating to this blog feature would look like: 

```markdown
src
├── content
│   ├── test-md
│   │   ├── test-md.md
│   │   └── test-png.png
│   └── example-blog
│       ├── example-blog.md
│       └── bird.png
├── pages
│   └── blog
│       ├── index.js
│       └── [slug].js
├── lib
│   └── blogmd.js
└── bin
    └── copy-images.mjs
```

As you see, inside `/content` folder are a bunch of folders of "blog post" named intended for the url slug, such as `/content/test-md` or `/content/example-blog`. Inside each folder we have the markdown file with the same name. We also have a bunch of images that we referenced in our markdown file.

In our `/pages` folder, you would possibly have some of your own pages written up. But we need to add the `/pages/blog` folder which we are going to add `/pages/blog/index.js` the index page and `/pages/blog/[slug].js` the url slug pages by [Dynamic Routing](https://nextjs.org/docs/routing/dynamic-routes).

Below we have `/lib/blogmd.js` which contains some helper functions that reads and parses our markdown files. As well as `/bin/copy-images.mjs` which copies our images of each blog to the `/public` folder for Next.js to use when built and deployed.

## Markdown Content

1. Go ahead and create some folders of markdown files under `/content` folder. For example, we can create `/content/test-md/test-md.md` for testing purpose.

If you are new to markdown, take a look at the [Markdown Guide](https://www.markdownguide.org). And maybe open the [Cheat Sheet](https://www.markdownguide.org/cheat-sheet/) on the side when you are writing in .md just as I am doing now.

```markdown
---
title: 'The First Test Blog in Markdown'
publishedAt: 2022-4-12
---

This is a test blog written in md.

Here is a list:
- Something
- Something else

> This is a quote
```

2. In order to read the markdown files, we need to update the `next.config.js` to include .md files:

```jsx
// next.config.js

/** @type {import('next').NextConfig} */
const nextConfig = {
  // Configure pageExtensions to include md and mdx
  pageExtensions: ['ts', 'tsx', 'js', 'jsx', 'md', 'mdx'],
  reactStrictMode: true,
}

module.exports = nextConfig
```

## Helper Functions and Blog Pages

Now it is the time to create the Blog Index Page and slug pages for each blog markdown files. But before that, we need some helpers to parse the markdown files. To do that, we need [gray-matter](https://github.com/jonschlinkert/gray-matter) to parse front matter and convert YAML into an object.

1. Go ahead and install gray-matter:

```jsx
npm install gray-matter
```

2. Create the `/lib/blogmd.js` file and add some of our helper functions:

```jsx
const fs = require("fs");
const path = require("path");
const matter = require('gray-matter');

// Get full path to the content directory of blogs folders, in this case /src/content
export function getPath(contentFolder) {
  return path.join(process.cwd(), `/${contentFolder}`);
}

// Get the markdown file data from its parent folder with the same name in the content directory
export function getFileContent(folderName, contentFolder) {
  const BLOGS_DIRS_PATH = getPath(contentFolder);

  // Create the full path to the .md file
  const filePath = path.join(BLOGS_DIRS_PATH, folderName, `${folderName}.md`);
  const fileData = fs.readFileSync(filePath, "utf-8");
  return fileData;
}
```

`getPath(contentFolder)` will create the full path to the content folder to use later, in our case, whenever we call this function we should call it with 'src/content'.

Notice `process.cwd()` returns the [current working folder](https://nodejs.org/api/process.html#processcwd) of the Node.js process when the website is deployed on a server service of your choice.

`getFileContent(folderName, contentFolder)` function will take both the slug name of the blog markdown file (also its parent folder's name), and again `contentFolder` ('src/content'). It will then read the file and return the file data for us to process next.

3. To process the markdown file data, we will now add this function to `/lib/blogmd.js`:

```jsx
// Get a single blog from the content folder with the given slug
export function getSingleBlog(slug, contentFolder) {
  const source = getFileContent(slug, contentFolder);
  const { data, content } = matter(source);
  return {
    meta: data,
    content: content,
  }
}
```

This function uses `matter(source)` to parse the data and return an object including:

- `data`: the meta data at the top of our markdown file, includes `title` and `publishedAt` in this case.
- `content`: the content of the markdown file.

4. Then we add another function to read and return an array of blogs data:

```jsx
// Get all blogs from the content folder
export function getAllBlogs(contentFolder) {
  const BLOGS_DIRS_PATH = getPath(contentFolder);

  // Get a list of all blog folders' names
  let folderNames = fs.readdirSync(BLOGS_DIRS_PATH);

  // Filter out the hidden files, this relates to MacOS system
  folderNames = folderNames.filter((fileName) => !fileName.startsWith('.'));

  // Using gray-matter to parse the markdown file and return meta, content and slug
  const blogs = folderNames.map((folderName) => {
    const source = getFileContent(folderName, contentFolder);
    const { data, content } = matter(source);
    return {
      meta: data,
      content: content,
      slug: folderName,
    }
  });
  return blogs;
}
```

Now whenever we call `getAllBlogs(contentFolder)`, we will get an array of objects which contains all of the data we parsed from our markdown files, including `meta`: to use for titles or published dates, `content`: to render the main content and `slug`: to use in url slugs.

## Blog Pages

I am sure you have basic knowledge of creating [pages](https://nextjs.org/docs/basic-features/pages) in Next.js as well as [Dynamic Routes](https://nextjs.org/docs/routing/dynamic-routes). We are going to do exactly that.

### Blog Index Page

First let's create the Blog Index Page at `base_url/blog`, where a list of blog titles and date published are displayed:

We are going to use [Static Site Generation](https://nextjs.org/docs/basic-features/data-fetching/get-static-props) for the blog index page. 

1. Go ahead and create `src/pages/blog/index.js` for the blog index page.

```jsx
import { getAllBlogs } from "@/lib/blogmd";

export async function getStaticProps() {
  const blogs = getAllBlogs('src/content');
  blogs.sort((a, b) => {
    if ((new Date(a.meta.publishedAt)) > new Date(b.meta.publishedAt)) {
      return -1;
    }
    return 1;
  });

  return {
    props: {
      blogs: blogs,
    }
  }
}
```

Inside `getStaticProps()` function, we will call our helper function `getAllBlogs(contentFolder)` and then sort the blogs by descending order of the published date from `meta.publishedAt`. Note that the function returns `blogs` in props, which is an array of blog data for the next BlogPage function to use.

2. Then we need to write the main function for our index page:

```jsx
import Link from "next/link";

export default function BlogPage({ blogs }) {
  return (
      <div>
        <h1>Blog</h1>
        <ul>
          {blogs.map((blog) => (
            <li key={blog.slug}>
              <Link href={`/blog/${blog.slug}`}>
                <h2>{blog.meta.title}</h2>
                <span>{blog.meta.publishedAt}</span>
              </Link>
          </li>
          ))}
        </ul>
      </div>
  );
}
```

This is a simple function where an unordered list of `<Link>` to each blog post are displayed. We use `blog.slug` for the url of individual blog page, e.g. `base_url/blog/test-md` would now render out the test-md.md content.

### Blog [slug] Page

Since we have the blog index page to redirect to each blog post, now we need to create the blog pages with some of Next.js's amazing features.

1. Create `src/pages/blog/[slug].js` and add the following functions:

```jsx
import { getAllBlogs, getSingleBlog } from '@/lib/blogmd';

export async function getStaticPaths() {
  const blogs = getAllBlogs('src/content');
  const paths = blogs.map((blog) => ({
    params: {
      slug: blog.slug,
    },
  }))
  return {
    paths,
    fallback: false,
  }
}

export async function getStaticProps(context) {
  const blog = getSingleBlog(context.params.slug, 'src/content');
  return {
    props: {
      blog: blog
    },
  }
}
```

For someone with Next.js experience, this would be easy to understand. 

`getStaticPaths()` function will define a [list of paths](https://nextjs.org/docs/basic-features/data-fetching/get-static-paths) that will be statically generated. In our case, we call `getAllBlogs(contentFolder)` and get an array of blogs and we will create our paths from `blog.slug`, as it matches the redirect url from our blog index page's links.

`getStaticProps()` will use the slug to then call `getSingleBlog(folderName, contentFolder)`, which just to remind you, takes two parameters: `folderName` the name of the blog folder, and `contentFolder` the parent folder of these blog folders. In our case we call `getSingleBlog(slug, 'src/content')` to get the blog data.

2. Now we need to install [react-markdown](https://github.com/remarkjs/react-markdown), to take a string of markdown and it’ll safely render to React elements. The benefits are in their documentation and you can read about it too.

```jsx
npm install react-markdown
```

3. Then let's write our Blog page function, don't forget to import `ReactMarkdown` too.

```jsx
export default function Blog({ blog }) {
  return (
    <article>
      <h1>{blog.meta.title}</h1>
      <p>{blog.meta.publishedAt}</p>
      <ReactMarkdown>{blog.content}</ReactMarkdown>
    </article>
  );
}
```

Again a simple function to understand, this `<article>` will include our title, published date and the main content which is rendered by `<ReactMarkdown>`. Just to recall, in our `getSingleBlog()` helper function, it will return these data after parsing with [gray-matter](https://github.com/jonschlinkert/gray-matter), which we used in the blog function.

```jsx
export function getSingleBlog(slug, contentFolder) {
  const source = getFileContent(slug, contentFolder);
  const { data, content } = matter(source);
  return {
    meta: data,
    content: content,
  }
}
```

4. Now your markdown blog is rendered correctly! Great!

## Styling

Now you are probably thinking about styling your blog pages. You can go ahead and use CSS or any frameworks you like to style the components of your markdown blog.

What I use is [Tailwind Typography](https://tailwindcss.com/docs/typography-plugin) plugin which automatically style the HTML elements from my markdown just by adding `class="prose"` to my `<article>`.

Give their documentations a read and have a try. It looks nice and you can add [Dark Mode](https://tailwindcss.com/docs/typography-plugin#adapting-to-dark-mode) super easily too.

## Images in Markdown

Adding images in markdown file is easy, just like so:

```markdown
![test-png](/blogImages/test-md/test-png.png)
```

Note that the base url of the image needs to be the `public` folder, according to [Static File Serving](https://nextjs.org/docs/basic-features/static-file-serving). 

But it would be extremely painful for us to stay in the flow of writing our markdown content, if we need to add the image to the corresponding blog folder inside the public folder (if you want it well structured), e.g. `public/blogImages/test-md`.

Therefore let's write a Node.js script that is going to automatically copy our images from `src/content/test-md` to `public/blogImages/test-md` when the website is built and deployed:

1. Go ahead and create `src/bin/copy-images.mjs`, and add the following functions:

```jsx
import fs from 'fs';
import path from 'path';
import fsExtra from 'fs-extra';

const fsPromises = fs.promises;
const targetDir = path.join(process.cwd(), 'public/blogImages');
const blogsDir = path.join(process.cwd(), 'src/content');

async function createPostImageFoldersForCopy() {
  // Get every blog folder: blog-one, blog-two etc.
  let blogsSlugs = await fsPromises.readdir(blogsDir);
  
  // Get rid of MacOS's .DS_Store hidden file
  blogsSlugs = blogsSlugs.filter((slug) => !slug.startsWith('.'));

  for (const slug of blogsSlugs) {
    const allowedImageFileExtensions = ['.png', '.jpg', '.jpeg', '.gif'];

    // Read all files inside current blog folder
    const blogDirFiles = await fsPromises.readdir(`${blogsDir}/${slug}`);

    // Filter out files with allowed file extension (images)
    const images = blogDirFiles.filter(file =>
      allowedImageFileExtensions.includes(path.extname(file)),
    );

    if (images.length) {
      // Create a folder for images of this post inside public
      await fsPromises.mkdir(`${targetDir}/${slug}`);

      await copyImagesToPublic(images, slug);
    }
  }
}

async function copyImagesToPublic(images, slug) {
  for (const image of images) {
    await fsPromises.copyFile(
      `${blogsDir}/${slug}/${image}`,
      `${targetDir}/${slug}/${image}`
    );
  }
}

await fsExtra.emptyDir(targetDir);
await createPostImageFoldersForCopy();
```

The functions in this script are basically reading allowed files (png, jpg, jpeg, gif) inside every blog folder, and copy them to a corresponding folder in the targeted folder for blog images in the public folder.

2. Then update your `package.json` so the script runs during the [pre-build](https://docs.npmjs.com/cli/v9/using-npm/scripts#pre--post-scripts) stage:

```jsxon
"scripts": {
  "copyimages": "node ./src/bin/copy-images.mjs",
  "prebuild": "npm run copyimages",
  "dev": "next dev",
  "build": "next build",
}
```

3. Now if you run `npm run build`, the images will be copied to the public folder automatically.

## Conclusion

From when I started trying Next.js, I liked it immediately for its various features and library content. 

I am sure there are a lot of ways of implementing a Markdown blog feature in your Next.js project. But these are what I have come up with after going through a lot of readings and documentations. 

Hopefully you learned something from this blog, written in Markdown. 😎

---

Possible updates:
- Adding [react-syntax-highlighter](https://github.com/react-syntax-highlighter/react-syntax-highlighter) for code block styles.
- Creating a repo of a template of this blog so it can work right off the start.

---

This blog post is also published on Medium [here](https://luuu-xu.medium.com/how-to-add-markdown-blog-feature-to-your-next-js-website-c036ad4d17d7). 

[How to Add Markdown Blog Feature to Your Next.js Website](https://luuu-xu.medium.com/how-to-add-markdown-blog-feature-to-your-next-js-website-c036ad4d17d7)

During my self-learning journey of programming, I have read a ton of Medium articles and they all more or less helped me. These posts are actually the most important force pushing me writing this blog and I appreciate them a lot!

---

Photo by [Laura Chouette](https://unsplash.com/@laurachouette?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/photos/6tdfx_gvHf0?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText).


I was checking my [Odinbook](https://odinbook-client-production-9219.up.railway.app/) (Facebook-clone) website after deployment, where some posts and comments were made by nice fellow users on [The Odin Project](https://www.theodinproject.com), I found an error in one post content:

![&#x27; error](/blogImages/remove-escape/user_post_escape_error.png)

Quickly I realized the `&#x27;` is essentially `'` character, which was escaped by [express-validator](https://express-validator.github.io/docs) in my Node.js middleware that was validating and sanitizing user inputs.

The function in questioning is [`escape(input)`](https://github.com/validatorjs/validator.js#sanitizers#escape), from the documentation, it replace **<, >, &, ', " and /** with HTML entities.

Reminding of myself, I used `escape()` to sanitize the user input because I was trying to prevent **cross-site-scripting (XSS)** attacks. The idea of an XSS attack sounded pretty serious to me at the time of introduction, later I would sanitize user inputs in every Node.js back-end controllers. 

An example of XSS attack would look like:
1. App allows user to input post content which would be displayed after submitting.
2. Attacker enters the input `attacker <script>alert('You are being attacked!')</script>`.
3. Now the post content would render the `script` which annoyingly alerts the attack.

By escaping such user inputs first, then storing into database (my choice being MongoDB), I had the idea that it was successfully preventing XSS attacks. That's up until Visitor-67's asking for a partner for JavaScript30 course post came along.

Now I had solutions in mind, maybe I could unescape characters other than `<>` on front-end? But what if a user is trying to post some content about HTML where `<>` are needed? What about not escaping user inputs at all? But what if some nasty attacks actually gets stored and rendered on my website?

Just when I was researching on this topic, a post by Ben Hoyt came along, [Don’t try to sanitize input. Escape output.](https://benhoyt.com/writings/dont-sanitize-do-escape/). The post made sense to me about the idea of "escaping output instead" especially with the quote:

> In short, it’s no good to strip out “dangerous characters”, because some characters are dangerous in some contexts and perfectly safe in others.

And this was when I realized that most front-end frameworks like React has "contextual escaping" in box, where escaping are done before outputting HTML, as well as JSON.

Great! Now I could feel safe removing `escape()` sanitizer from all of my controllers dealing with user inputs. 

But the researches brought another issue to my attention, which was **NoSQL injection** especially towards MongoDB.

An example of MongoDB injection would look like:

1. When user is logging in, the controller would do something like this:
  ```js
  db.users.find({
    username: req.body.username,
    password: req.body.password
  });
  ```
2. An attacker inputs `{ $ne: null }` in password instead which is essentially querying a password that is not equal to null.
3. Now the attacker would be logged in.

Solutions? Yes, actually you can use [mongo-sanitize](https://github.com/vkarpov15/mongo-sanitize), which does one thing: "strip out any keys that start with '$' in the input" and it is exactly what you want.

Just before I was going to `npm install mongo-sanitize`, I found out that with [mongoose](https://mongoosejs.com/docs/guide.html), after defining your Schemas where values required SchemaType like string or boolean, the `{ $ne: null }` would be passed to the password query selector as plain strings. Therefore NoSQL injections are prevented.

Nice!

Until now, I felt the issue was fairly understood, looked into, and solved.

And that was the whole hour spent behind the decision to safely removing `escape()` sanitizer function from my backend controllers. knowing both XSS attacks and NoSQL injections are being prevented by my website.


This is a test blog written in md.

Here is a list:
- Something
- Something else

> This is a quote

```
const md = markdown;
```

![my github avatar](/blogImages/test-md/github-avatar.jpeg)

Blog

Designing Data-Intensive Applications - 2. Distributed Data

Implementing Boids Algorithm on Canvas

Designing Data-Intensive Applications - 1. Foundations of Data Systems

Clean Code and Leather-working

How to Add Markdown Blog Feature to Your Next.js Website

The Decision not to Sanitize User Inputs

The First Test Blog in Markdown