Why This News Matters
STCLab's migration to Open Telemetry has significant implications for the tech industry, particularly in North East India and broader India. By adopting open observability standards, STCLab has demonstrated a commitment to cost-effective, efficient, and flexible solutions, paving the way for other organizations to follow suit.
Challenges Faced and Solutions Implemented
In 2023, STCLab faced the daunting task of managing millions of concurrent connections during global events. To tackle this challenge, they decided to sunset their 20-year legacy, on-premise architecture and build a global, Kubernetes-native SaaS instead. The decision aimed to reduce costs and prevent performance regressions due to reactive firefighting.
Migrating to Open Telemetry
STCLab chose OpenTelemetry for instrumentation and the LGTM stack (Loki, Grafana, Tempo, and Mimir). This migration resulted in a 72% cost reduction compared to their previous vendor, 100% APM trace coverage in all environments (up from 5% sampled in production), and unified observability across all Kubernetes clusters.
Architectural Decisions and Key Configurations
Centralized Backend and Distributed Collectors
STCLab centralized all telemetry into a single management cluster using multi-tenancy rather than deploying full LGTM stacks everywhere. Each cluster deploys a lightweight OTel Collector that injects tenant IDs via X-Scope-OrgID header. A central backend of Mimir, Loki, and Tempo isolates data by tenant, preventing noisy neighbor issues.
OpenTelemetry as the Universal Ingestion Layer
The OTel Collector handles multi-tenancy tagging, batching, buffering, retries, and tail sampling. OTel auto-instrumentation for Java and Node.js workloads enables full APM without modifying application code, and complete backend decoupling can be achieved by changing one config line from Tempo to Jaeger.
Key Challenges and Solutions
The Metric Explosion
Deploying OTel Collector as a DaemonSet caused metrics to explode, leading to inefficiencies. STCLab resolved this issue by implementing a Target Allocator per-node strategy that assigns scrape jobs only to collectors on the same node as targets.
Version Alignment
Version alignment between the Operator, Collector, and Target Allocator is crucial to avoid issues like scrape pool failures. STCLab learned to explicitly unify all component versions to ensure smooth operation.
Small Node OOM
Collectors on 2GB nodes would OOM and hang entire nodes. To address this, STCLab deployed collectors only on nodes with at least 4GB of memory.
Reflections and Future Steps
STCLab's journey to Open Telemetry offers valuable insights for tech companies in North East India and beyond. By sharing their experiences, STCLab hopes to help other teams on a similar path, fostering a more efficient and cost-effective tech ecosystem.