Posts tagged with sre

PlanetScale Tech BlogJul 23, 2024

The State of Online Schema Migrations in MySQL

Why it matters: Understanding MySQL migration algorithms is crucial for maintaining high availability. Choosing the wrong method can lead to performance degradation, massive replication lag, or accidental data loss in high-traffic production environments where schema evolution is constant.

MySQL's INPLACE algorithm allows non-blocking DML on the primary but causes significant replication lag and high resource consumption.
INSTANT DDL is the preferred method for supported changes as it modifies metadata without table copies or replication delays.
Since MySQL 8.0.29, INSTANT supports arbitrary ADD COLUMN and DROP COLUMN operations, though limitations remain for columns in indexes.
Many common operations, like changing data types or adding indexes, are still unsupported by INSTANT and require alternative strategies.
DROP COLUMN via INSTANT carries risks of permanent data loss and breaking queries that reference the column explicitly.
Revertibility is a critical consideration; while ADD COLUMN is easily undone, DROP COLUMN requires schema history and data recovery.

#data #sre

Read original

PlanetScale Tech BlogJul 10, 2024

Dealing with large tables

Why it matters: Scaling databases is a critical challenge as applications grow. Understanding the transition from vertical scaling to vertical sharding helps engineers maintain performance and manage costs when single-node limits are reached, especially for high-growth tables like logs or activity feeds.

Rapidly growing tables, such as activity logs, often require specialized scaling strategies compared to smaller, stable reference tables.
Vertical scaling increases a single server's CPU, RAM, and disk, but becomes prohibitively expensive and performance-limited at the multi-terabyte scale.
Managed database platforms automate storage expansion and simplify compute upgrades, reducing the operational overhead of manual disk management.
Vertical sharding involves moving specific high-traffic tables to separate physical servers to isolate resource contention and improve scalability.
Tools like Vitess enable vertical sharding through workflows like MoveTables, allowing engineers to migrate data between keyspaces with minimal application disruption.

#data #dist #sre

Read original

Instagram EngineeringOct 17, 2019

Python at Scale: Strict Modules

Why it matters: Managing a multi-million line Python monolith requires addressing the risks of dynamic imports. Uncontrolled side effects and global state mutation slow down development cycles and introduce production instability, necessitating stricter module boundaries for performance and reliability.

Instagram's multi-million line Python monolith faces significant performance bottlenecks due to arbitrary code execution during module imports.
Import-time side effects like regex compilation and decorator execution prevent incremental reloading, causing server startup times of up to 60 seconds.
Unsafe import practices, such as fetching network configuration at the module level, lead to non-deterministic initialization failures and production risks.
The dynamic nature of Python allows for mutable global state, which often causes request pollution and test flakiness in large-scale environments.
Standard Python lacks explicit control over import order, making it difficult to prevent 'spooky action at a distance' bugs in complex dependency graphs.

#sre #culture

Read original

Page 23 of 23

Prev 1...21 22 23Next