Platform Setup

This repo contains guidelines and steps for setting up in-house production infrastructure and cloud services of open-source technologies from scratch.

Data Engineering Stack

• CDC Pipelines

  • High-performance coordination service for distributed applications:
    • Apache Zookeeper
  • Distributed event streaming platform:
    • Apache Kafka
  • Distributed framework to stream data into and out of Apache kafka:
    • Confluent Inc. Kafka Connect
  • Distributed registry to store kafka-payload's schemas:
    • Confluent Inc. Schema Registry

• Databases

  • SQL/RDBMS:
    • PostgreSQL
    • Timescale
    • Mysql
  • NoSQL
    • Document:
      • MongoDB
    • Key-value:
      • Redis
    • Graph:
      • Neo4j
    • Time Series:
      • Prometheus (NoSQL)
      • Timescale (SQL)

• Distributed Workflow Management

  • Apache Airflow

• Big Data

  • Distributed SQL Query Engine on any data storage:
    • Prestodb
    • PrestoSQL/Trino
  • Distributed & Resilient Data Processing framework:
    • Apache Spark
  • SQL on HDFS:
    • Apache Hive

• Search Engines

  • Elasticsearch

• Centralized Logging

  • Elastic Stack
    • Filebeat
    • Elasticsearch-Ingest-Pipeline
    • Kibana

• Business Intelligence

  • Apache Superset

Platform Engineering Stack

• Container Orchestration

  • Kubernetes

• Service Discovery, Health Checking & Configuration

  • Consul

• Service Monitoring

  • Grafana
  • Prometheus

• SSL Certs

  • Let's Encrypt

• Load balancing & Reverse Proxying

  • NGINX

• VPN

  • OpenVPN

• Linux

  • Linux Utilities

...