DEV Community

dima853
dima853

Posted on

2.3.2 - 2.3.5 Concepts (System Performance Brendan Gregg 2nd)

#programming #security #architecture #design

2.3.3 Trade-Offs

Trade-offs in IT: "Pick two" and performance optimization

The image illustrates the classic principle "Good, Fast, Cheap — choose two", adapted for IT projects:

  • High performance (High-Performance)
  • Deadlines (On-Time)
  • Budget (Inexpensive)

They usually choose "On-Time+ Inexpensive", sacrificing performance, which leads to problems:

  • Suboptimal storage architecture.
  • Inefficient programming languages.
  • Lack of tools for performance analysis.

1. Key trade-offs in IT

(1) CPU vs Memory

  • In-memory caching → reduces CPU load, but requires more RAM.
  • Data compression → saves memory, but increases CPU load.

Example:

  • Redis (in-memory cache) vs PostgreSQL (page compression).

(2) File System block Size

Small block (4 KB) Large block(64 KB)
Better for random I/O operations Better for streaming reads/writes
Uses cache more efficiently Speeds up backups

(3) Network Buffer size

Small buffer Large buffer
Less memory load Higher bandwidth

2. Where is it more efficient to adjust performance?

The closer the setting is to the application level, the greater the effect.:

Level Examples of optimizations Potential gain
Application Optimization of SQL queries and logic Up to 20x
Database Indexes, partitioning Up to 5x
File system Configuring the cache, block size Up to 2x
Storage RAID, SSD vs HDD Up to 1.5x

** Why?**

An application-level fix eliminates work at all lower levels of the stack.

3. Problems of modern approaches

  1. The race for functionality:

    • Developers focus on correctness, not performance.
    • Performance problems are detected already in production.

  2. Cloud environments:

  3. Frequent updates (weekly/daily) complicate long-term monitoring.

    • It is important to monitor the OS, even if the problem is in the application.

4. Practical advice

(1) For developers

  • Profile the code before release (for example, perf for Linux, VTune for Intel).
  • Avoid N+1 queries in databases.
  • Cache hot data (Redis, Memcached).

(2) For admins

  • Set up OS monitoring (Prometheus + Grafana for CPU/RAM/I/O metrics).
  • Optimize block sizes for load (for example, ext4 with bs=64k for DBMS).
  • Use modern storage (NVMe for high-load systems).

Conclusion

  1. Sacrifice one of the three: speed, quality, or cost.
  2. Optimize at the application level — this gives maximum effect.
  3. Monitor the OS, even if the problem seems to be "applied".

"Late optimization is the root of all evil. But even worse is blind optimization without measurements."

— Adaptation of a quote by Donald Knuth

Optimizing block size for PostgreSQL: how to choose and why?

The size of a block (page) in PostgreSQL critically affects performance. Let's figure out how to choose it correctly for your workload.

1. What is a "block" in PostgreSQL?

  • Standard size: 8 KB (can be changed from 1 to 32 KB during compilation).
  • One page = 1 block = minimum unit of I/O operations.
  • Stores: table rows, indexes, metadata.

2. How does block size affect performance?

Small block (4 KB) Large block (16-32 KB)
✅ Better for OLTP (lots of random reads/writes of short strings) ✅ Better for analytical queries (scanning large ranges)
❌ Accesses disk more often (less data per 1 I/O) ❌ Spends memory on empty areas if the rows are small

Example:

If you have a chat app (lots of inserts/updates of short messages) — 8 KB optimal.

For Data Warehouse (large table analytics) — 16-32 KB will speed up `SELECT * FROM large_table'.

3. How can I check my current performance?

Diagnostic requests:
`sql
-- Average row size in the table
SELECT avg(pg_column_size(t.*)) FROM my_table t;

-- How many blocks are "empty" (low occupancy)
SELECT relname, 100 - (pg_relation_size(oid) / (8192 * relpages) * 100) AS "empty_space_%"
FROM pg_class WHERE relkind = 'r';
`
If `empty_space_%' is > 30%, it may be worth reducing the block size.

4. How can I change the block size?

  1. Only when compiling PostgreSQL (setting --with-blocksize=16 in ./configure).
  2. Cannot be changed for an existing database — only create a new cluster with a different size: 
   initdb -D /path/to/new/data --block-size=16384
Enter fullscreen mode Exit fullscreen mode
  1. Data transfer: Use pg_dump/`pg_restore'.

5. Optimal values for different scenarios

Load type Recommended unit Arguments
OLTP (short transactions) 8 KB Minimizing overhead
Analytics (large scans) 16-32 KB Reducing the number of I/O operations
Mixed load 8 KB Read/Write balance

6. What else affects I/O efficiency?

  • TOAST: PostgreSQL automatically compresses/separates large fields (> 2 KB).
  • Fillfactor: Setting the page occupancy (for example, 90 for frequently updated tables).
  • File system: 'XFS or ZFS with the setting bs=16k` to match PostgreSQL.

Conclusion

  1. 8 KB is a safe choice for most cases.
  2. 16+ KB — only for post-test analytics.
  3. Before changing: Measure the current efficiency via 'pg_stat_io'.

Tip: For cloud databases (RDS, Cloud SQL), the block size is fixed — choose instances with preset values for your load.

The level of depth of performance analysis: when to stop?

Productivity is a balance between analysis costs and potential benefits. Let's look at how to determine the optimal level of effort for your project.

1. "Adequate level" of analysis: from startups to HFT

Organizations have different approaches to performance analysis depending on ROI (return on investment):

Company type The level of analysis Examples
Startups Surface monitoring (CloudWatch, Sentry) Checking API latency and alerts
Corporations Deep analysis (perf, eBPF, CPU PMC) Linux kernel optimization, simulation
HFT / Exchanges Extreme optimization (nanoseconds) Laying cables for $300 million to save 6 ms

Key principle: > "Adequate level" is when ** the cost of the analysis is less than the potential benefit**.

2. When should I stop the analysis?

Scenarios for stopping:

  1. ** The main reason is explained**

  2. Example: A Java application eats CPU. We found exceptions that explain 12% of the load. But since slowdown 3x, we're looking further.

    • Rule: We stop if we find >66% problems.

  3. ROI is less than the cost of analysis

  4. Optimization of a microservice that saves $100/year is not worth 10 hours of an engineer's work.

    • Exception: If the problem is a "canary" for future disasters (for example, memory leak).

  5. There are more important tasks

    • Even if the problem is not completely clear, there are critical bugs elsewhere.

3. Recommendations "at a given time"

Performance settings ** become obsolete** after:

  • Hardware upgrades (for example, switching from 10 Gbit/s to 100 Gbit/s).
  • Software updates (the new version of PostgreSQL may change the optimal `shared_buffers').
  • Growth in the number of users.

How to avoid mistakes:

  • Do not copy tunings from the Internet blindly. Example: nginx # Old recommendation for Linux 2.6: net.ipv4.tcp_tw_reuse = 1 # May be harmful in modern kernels!
  • Version the settings (Git + annotations like: "Increased vm.swappiness=10 due to swap load in 2023-01").

4. Practical rules

For startups

  • Minimum: Latency/error monitoring (New Relic, Datadog).
  • Optimum: Once a quarter — deep check of slow queries and caching.

For corporations

  • Permanent team of performance engineers.
  • Tools:
  • perf/`eBPF' — for core analysis.
    • Load forecasting (for example, via ML).

For HFT/Exchanges

  • Hard SLA: Every microsecond is money.
  • Investments: Custom network stacks, FPGA instead of CPU.

5. Checklist "Where to stay?"

  1. Have you found the cause of >50% of the problem? → Stop, we're fixing.
  2. Do the remaining hypotheses take a disproportionate amount of time? → Postpone.
  3. Are there more important tasks with high ROI? → Switch to.

Metric: If the analysis takes longer than the potential savings, you have overdone it.

Conclusion

  • The depth of the analysis depends on budget and risks.
  • Tuning is outdated — document the changes.
  • 80/20 rule: 20% of efforts give 80% of the result.

Case study:
After updating the AWS instance from m5.large to m6i.xlarge', the oldkernel.schedul_migration_cost_ns` settings began to slow down the application. Solution: reset the parameters to default + measurements under load.

Top comments (0)