Posts tagged with data

PlanetScale Tech BlogJul 10, 2024

Why it matters: Scaling databases is a critical challenge as applications grow. Understanding the transition from vertical scaling to vertical sharding helps engineers maintain performance and manage costs when single-node limits are reached, especially for high-growth tables like logs or activity feeds.

Rapidly growing tables, such as activity logs, often require specialized scaling strategies compared to smaller, stable reference tables.
Vertical scaling increases a single server's CPU, RAM, and disk, but becomes prohibitively expensive and performance-limited at the multi-terabyte scale.
Managed database platforms automate storage expansion and simplify compute upgrades, reducing the operational overhead of manual disk management.
Vertical sharding involves moving specific high-traffic tables to separate physical servers to isolate resource contention and improve scalability.
Tools like Vitess enable vertical sharding through workflows like MoveTables, allowing engineers to migrate data between keyspaces with minimal application disruption.

#data #dist #sre

Read original

PlanetScale Tech BlogJul 8, 2024

Sharding strategies: directory-based, range-based, and hash-based

Why it matters: Understanding sharding strategies is crucial for scaling databases effectively. Choosing the right approach prevents hotspots, ensures even data distribution, and minimizes latency, which are critical factors for maintaining high-performance distributed systems as data volume grows.

Sharding distributes database data across multiple instances using a shard key to determine placement.
Directory-based sharding uses a lookup table for mapping, offering flexibility but introducing extra query latency and potential hotspots.
Range-based sharding divides data by value intervals, which is simple to implement but often leads to unbalanced shards due to uneven data frequency.
Hash-based sharding applies a hash function to the shard key to ensure uniform data distribution and is generally recommended as the default strategy.
The choice of strategy impacts system performance, specifically regarding data access patterns, resharding complexity, and load balancing.

#data #dist

Read original

Instagram EngineeringJan 25, 2020

Five things I learned about working on content quality at Instagram

Why it matters: Managing content quality at scale requires balancing real-time signals with static analysis. This approach shows how to operationalize quality metrics and use multi-stage ML pipelines to protect users while maintaining high-performance recommendation systems.

Combined manual labeling with classifier scores to create calibrated metrics for statistically significant A/B testing results.
Developed 'read-path' models that utilize real-time engagement signals like comments and likes to improve detection precision.
Maintained 'write-path' filters at the sourcing level to handle low-prevalence violations and ensure a baseline of benign content.
Implemented a multi-stage pipeline that balances high-precision sourcing filters with fine-tuned ranking models.
Established continuous model performance tracking to identify edge cases and maintain user safety standards.

#mlp #data #security

Read original

Instagram EngineeringDec 13, 2019

Instagram Data Saver Mode

Why it matters: Engineers must balance performance and resource consumption. This case study shows how optimizing data usage through prefetching and resolution controls can improve user engagement and retention in data-constrained markets, proving that efficiency and growth can go hand-in-hand.

Instagram launched Data Saver Mode for Android to address high data consumption and improve efficiency relative to other Meta apps.
The implementation focuses on three levers: disabling video prefetch, disabling video autoplay, and offering manual media resolution controls.
Disabling prefetch ensures video data is only downloaded when a user stops scrolling, preventing waste on unviewed content.
Users can configure high-resolution media settings to 'Never,' 'Wi-Fi Only,' or 'Cellular and Wi-Fi' to manage their data budgets.
Global A/B testing showed that reducing data usage led to unexpected increases in user interactions and content creation.
The custom solution provides a smoother experience than Android's native Data Saver, which often blocks media loading entirely.

#mobile #data

Read original

Instagram EngineeringNov 26, 2019

Why it matters: This article provides a blueprint for building massive-scale recommendation engines. It demonstrates how custom DSLs and multi-stage filtering balance high-velocity experimentation with the extreme computational efficiency required to serve millions of users in real-time.

Instagram uses a three-stage ranking funnel to filter billions of media items into a personalized feed for each user in real-time.
Engineers developed IGQL, a C++ optimized domain-specific language, to allow for high-level algorithm design with low-latency execution.
The system utilizes 'ig2vec' account embeddings to identify topical similarities based on user interaction sequences, similar to word2vec.
Facebook’s FAISS library is used for efficient nearest-neighbor retrieval across millions of account embeddings.
The infrastructure supports massive scale, processing 65 billion features and making 90 million model predictions every second.

#mlp #data #dist

Read original

Instagram EngineeringOct 11, 2019

Making instagram.com faster: Part 3 — cache first

Why it matters: Cache-first rendering provides immediate UI feedback but creates complex state sync challenges. This approach shows how to use Git-like rebase patterns in Redux to ensure user interactions aren't lost when merging stale cached data with fresh server responses.

Implemented cache-first rendering by storing a subset of the Redux store in IndexedDB to allow immediate page hydration.
Addressed race conditions where user interactions on cached data, such as likes or comments, could be overwritten by incoming server responses.
Developed a staging mechanism that treats cached state as a local branch and server data as master, performing a rebase-like operation for state updates.
Created a staging API using stagingAction and stagingCommit to queue dispatched actions while network requests are pending.
Used a Redux reducer enhancer to apply queued actions to the fresh server state before committing it to the main store.
Achieved significant performance gains, including a 2.5% improvement in feed display time and an 11% improvement in stories tray display time.

#frontend #data

Read original

Instagram EngineeringSep 24, 2019

Interview with Tamar Shapiro, Instagram’s Head of Analytics

Why it matters: This interview offers a look into how Instagram uses data science and experimentation to drive product strategy. It highlights the intersection of technical leadership, user-centric culture, and the professional development skills necessary to succeed in high-scale engineering organizations.

Tamar Shapiro leads Instagram's analytics team, overseeing data scientists and engineers focused on experimentation and data-driven product decisions.
A key project discussed is the 'private like counts' test, which aimed to shift user focus from quantity to quality of interactions.
Instagram's engineering culture is characterized as highly collaborative with a 'people first' value system centered on the user community.
Effective communication and context sharing are emphasized as vital for maintaining alignment in fast-paced development environments.
Career growth advice for engineers includes prioritizing networking outside immediate teams and building confidence to advocate for accomplishments.

#data #culture

Read original

Page 19 of 19

Prev 1...17 18 19Next