RawDigs Infrastructure Map¶

Complete overview of the RawDigs microservices architecture, service connections, and data flow for Infrastructure as Code (IaC) implementation.

🏗️ Architecture Overview¶

graph TB
    subgraph "Client Layer"
        BROWSER[Web Browser]
        MOBILE[Mobile App]
    end

    subgraph "Application Layer"
        APP[Laravel Application<br/>:80]
        VITE[Vite Dev Server<br/>:5173]
        HORIZON[Horizon Dashboard<br/>/monitoring/horizon]
    end

    subgraph "Primary Data Layer"
        PGSQL[(PostgreSQL<br/>:5432<br/>App Database)]
    end

    subgraph "Monitoring Data Layer"
        MYSQL[(MySQL<br/>:3307<br/>Monitoring DB)]
    end

    subgraph "Cache Layer"
        MEMCACHED[(Memcached<br/>:11211<br/>Application Cache)]
    end

    subgraph "Session & Queue Layer"
        REDIS[(Redis<br/>:6379)]
        REDIS_DB0[DB 0: Default/General]
        REDIS_DB1[DB 1: Cache Store]
        REDIS_DB2[DB 2: Session Cache]
        REDIS_DB3[DB 3: Queues]

        REDIS --> REDIS_DB0
        REDIS --> REDIS_DB1
        REDIS --> REDIS_DB2
        REDIS --> REDIS_DB3
    end

    subgraph "Message Broker Layer"
        RABBITMQ[RabbitMQ<br/>:5672<br/>Management :15672]
    end

    subgraph "Search Layer"
        MEILISEARCH[Meilisearch<br/>:7700<br/>Full-Text Search]
    end

    subgraph "Storage Layer"
        MINIO[MinIO S3<br/>:9000<br/>Console :8900<br/>Object Storage]
        R2[Cloudflare R2<br/>Production Storage]
    end

    subgraph "Monitoring & Debugging"
        TELESCOPE[Telescope<br/>/telescope<br/>Debug Tool]
        PULSE[Pulse<br/>/pulse<br/>Performance Monitor]
    end

    subgraph "Security & Secrets"
        VAULT[HashiCorp Vault<br/>:8200<br/>Secrets Management]
    end

    subgraph "Development Tools"
        MAILPIT[Mailpit<br/>:8025<br/>Email Testing]
        DOCS[MkDocs<br/>:8000<br/>Documentation]
    end

    subgraph "External Microservices"
        AUDIO[Python Audio Service<br/>Audio Processing]
    end

    %% Client Connections
    BROWSER --> APP
    BROWSER --> VITE
    MOBILE --> APP

    %% App to Databases
    APP -->|Read/Write| PGSQL
    APP -->|Monitoring Data| MYSQL

    %% App to Cache/Session
    APP -->|Cache Ops| MEMCACHED
    APP -->|Sessions| REDIS_DB2
    APP -->|Queue Jobs| REDIS_DB3
    APP -->|General| REDIS_DB0

    %% App to Message Broker
    APP -->|Publish Events| RABBITMQ
    RABBITMQ -->|Consume Events| AUDIO

    %% App to Search
    APP -->|Index/Search| MEILISEARCH

    %% App to Storage
    APP -->|Store Files| MINIO
    APP -->|Production| R2
    AUDIO -->|Process Files| R2

    %% App to Monitoring
    TELESCOPE -->|Query| MYSQL
    PULSE -->|Query| MYSQL
    APP -->|Log Events| TELESCOPE
    APP -->|Record Metrics| PULSE

    %% App to Security
    APP -->|Fetch Secrets| VAULT

    %% Queue Processing
    HORIZON -->|Monitor| REDIS_DB3
    APP -->|Process| REDIS_DB3

    %% Dev Tools
    APP -->|Send Email| MAILPIT

    style APP fill:#4F46E5,stroke:#312E81,color:#fff
    style PGSQL fill:#336791,stroke:#1F4662,color:#fff
    style MYSQL fill:#00758F,stroke:#004D66,color:#fff
    style REDIS fill:#DC382D,stroke:#A6241C,color:#fff
    style RABBITMQ fill:#FF6600,stroke:#CC5200,color:#fff
    style VAULT fill:#FFD814,stroke:#CC9900,color:#000

🔌 Service Connection Matrix¶

📊 Data Flow Diagrams¶

User Request Flow¶

sequenceDiagram
    participant Browser
    participant Laravel
    participant PostgreSQL
    participant Redis
    participant Memcached

    Browser->>Laravel: HTTP Request
    Laravel->>Redis: Check Session (DB 2)
    Redis-->>Laravel: Session Data

    Laravel->>Memcached: Check Cache
    alt Cache Hit
        Memcached-->>Laravel: Cached Data
    else Cache Miss
        Laravel->>PostgreSQL: Query Database
        PostgreSQL-->>Laravel: Data
        Laravel->>Memcached: Store in Cache
    end

    Laravel-->>Browser: HTTP Response

Job Queue Flow¶

sequenceDiagram
    participant Laravel
    participant Redis
    participant Horizon
    participant Worker

    Laravel->>Redis: Dispatch Job (DB 3)
    Redis-->>Laravel: Job ID

    Horizon->>Redis: Monitor Queue
    Redis-->>Horizon: Queue Stats

    Worker->>Redis: Fetch Job
    Redis-->>Worker: Job Data
    Worker->>Worker: Process Job
    Worker->>Redis: Mark Complete

Media Upload & Processing Flow¶

sequenceDiagram
    participant User
    participant Laravel
    participant MinIO
    participant RabbitMQ
    participant AudioService
    participant R2

    User->>Laravel: Upload Audio File
    Laravel->>MinIO: Store Raw File (Dev)
    Laravel->>PostgreSQL: Create Media Record
    Laravel->>RabbitMQ: Publish media.uploaded Event

    RabbitMQ->>AudioService: Consume Event
    AudioService->>R2: Download Raw File
    AudioService->>AudioService: Process Audio<br/>(Transcode, Analyze, Generate Waveform)
    AudioService->>R2: Upload Processed Files
    AudioService->>RabbitMQ: Publish media.processed Event

    RabbitMQ->>Laravel: Consume Event
    Laravel->>PostgreSQL: Update Media Record
    Laravel-->>User: Processing Complete Notification

Monitoring Data Flow¶

sequenceDiagram
    participant Laravel
    participant Telescope
    participant Pulse
    participant MySQL
    participant Dashboard

    Laravel->>Telescope: Log Request/Query/Exception
    Telescope->>MySQL: Store Debug Entry (DB: monitoring)

    Laravel->>Pulse: Record Metric (Cache Hit, Slow Query, etc.)
    Pulse->>MySQL: Store Metric (DB: monitoring)

    Dashboard->>Telescope: View Debug Data
    Telescope->>MySQL: Query Entries
    MySQL-->>Dashboard: Debug Information

    Dashboard->>Pulse: View Metrics
    Pulse->>MySQL: Query Aggregates
    MySQL-->>Dashboard: Performance Data

🗄️ Database Architecture¶

PostgreSQL (Primary Database - Port 5432)¶

┌─────────────────────────────────────────┐
│         PostgreSQL Database             │
├─────────────────────────────────────────┤
│  Schema: public                         │
│                                         │
│  ✓ users                                │
│  ✓ roles                                │
│  ✓ permissions                          │
│  ✓ media                                │
│  ✓ jobs                                 │
│  ✓ failed_jobs                          │
│  ✓ sessions                             │
│  ✓ cache / cache_locks                  │
│  ✓ migrations                           │
│                                         │
│  Total Tables: 15                       │
│  Size: ~320 KB                          │
└─────────────────────────────────────────┘

MySQL Monitoring (Port 3307)¶

┌─────────────────────────────────────────┐
│      MySQL Monitoring Database          │
├─────────────────────────────────────────┤
│  Database: monitoring                   │
│                                         │
│  Telescope Tables:                      │
│  ✓ telescope_entries                    │
│  ✓ telescope_entries_tags               │
│  ✓ telescope_monitoring                 │
│                                         │
│  Pulse Tables:                          │
│  ✓ pulse_values                         │
│  ✓ pulse_entries                        │
│  ✓ pulse_aggregates                     │
│                                         │
│  Total Tables: 6                        │
│  Size: ~192 KB                          │
└─────────────────────────────────────────┘

Redis Database Layout (Port 6379)¶

┌─────────────────────────────────────────┐
│           Redis Databases               │
├─────────────────────────────────────────┤
│  DB 0: Default / General                │
│  ├─ Prefix: rawdigs_                    │
│  └─ Purpose: General cache, locks       │
│                                         │
│  DB 1: Cache Store                      │
│  ├─ Prefix: cache:                      │
│  └─ Purpose: Cache tags, locks          │
│                                         │
│  DB 2: Session Cache                    │
│  ├─ Prefix: session:                    │
│  ├─ Purpose: Session cache layer        │
│  └─ Source of Truth: PostgreSQL         │
│                                         │
│  DB 3: Queues                           │
│  ├─ Prefix: queue:                      │
│  └─ Purpose: Job queues, Horizon        │
└─────────────────────────────────────────┘

Note: Sessions use a database-first architecture with Redis as an optional cache layer: - Primary Storage: PostgreSQL sessions table (always reliable) - Cache Layer: Redis DB 2 (for performance) - Automatic Fallback: If Redis fails, sessions continue working from database

🔐 Security Architecture¶

graph LR
    subgraph "Secret Sources"
        VAULT[HashiCorp Vault<br/>:8200]
        ENV[.env File<br/>Development]
    end

    subgraph "Application"
        APP[Laravel App]
        CONFIG[Config Files]
    end

    subgraph "Secrets"
        DB_CREDS[Database Credentials]
        API_KEYS[API Keys]
        TOKENS[Auth Tokens]
        ENCRYPT[Encryption Keys]
    end

    ENV -->|Development| CONFIG
    VAULT -->|Production| CONFIG
    CONFIG --> APP

    APP --> DB_CREDS
    APP --> API_KEYS
    APP --> TOKENS
    APP --> ENCRYPT

    style VAULT fill:#FFD814,stroke:#CC9900,color:#000
    style APP fill:#4F46E5,stroke:#312E81,color:#fff

Vault Secret Paths¶

secret/
├── development/
│   ├── app              # APP_KEY, APP_SECRET
│   ├── database         # DB credentials
│   └── services         # Third-party API keys
├── staging/
│   ├── app
│   ├── database
│   └── services
└── production/
    ├── app
    ├── database
    └── services

🎯 Service Endpoints & Access¶

🚀 Environment Variables Map¶

Application Layer¶

APP_URL=http://localhost              → Laravel base URL
VITE_PORT=5173                        → Vite dev server

Database Layer¶

DB_CONNECTION=pgsql                   → PostgreSQL (primary)
DB_HOST=pgsql                         → Service name
DB_PORT=5432                          → PostgreSQL port
DB_DATABASE=laravel                   → Database name

DB_MONITORING_HOST=mysql_monitoring   → MySQL monitoring service
DB_MONITORING_PORT=3306               → MySQL port (exposed as 3307)
DB_MONITORING_DATABASE=monitoring     → Monitoring database

Cache & Session Layer¶

CACHE_STORE=memcached                 → Primary cache
MEMCACHED_HOST=memcached              → Memcached service
MEMCACHED_PORT=11211                  → Memcached port

# Sessions: Database-first with Redis cache
SESSION_DRIVER=database               → Database as source of truth
SESSION_CONNECTION=pgsql              → PostgreSQL connection
SESSION_TABLE=sessions                → Sessions table
SESSION_STORE=redis                   → Redis cache layer (optional)
REDIS_HOST=redis                      → Redis service
REDIS_PORT=6379                       → Redis port
REDIS_SESSION_DB=2                    → Redis database for session cache

Queue Layer¶

QUEUE_CONNECTION=redis                → Queue driver
REDIS_QUEUE_DB=3                      → Redis database for queues
HORIZON_PATH=monitoring/horizon       → Horizon URL path
HORIZON_NAME=RawDigs                  → Horizon instance name

Message Broker¶

RABBITMQ_HOST=rabbitmq                → RabbitMQ service
RABBITMQ_PORT=5672                    → AMQP port
RABBITMQ_USER=guest                   → Username
RABBITMQ_PASSWORD=guest               → Password

Search¶

SCOUT_DRIVER=meilisearch              → Search engine
MEILISEARCH_HOST=http://meilisearch:7700  → Meilisearch URL

Storage¶

FILESYSTEM_DISK=local                 → Dev storage
# Production: s3 or r2

Monitoring¶

TELESCOPE_ENABLED=true                → Enable Telescope
TELESCOPE_DB_CONNECTION=mysql_monitoring  → Telescope database
PULSE_ENABLED=true                    → Enable Pulse
PULSE_DB_CONNECTION=mysql_monitoring  → Pulse database

Security¶

VAULT_ADDR=http://vault:8200          → Vault URL
VAULT_ROOT_TOKEN=xxx                  → Vault token

📦 Docker Services¶

Service Dependencies¶

graph TB
    APP[laravel.test]
    HORIZON[horizon]

    APP --> PGSQL
    APP --> MYSQL
    APP --> REDIS
    APP --> MEMCACHED
    APP --> MEILISEARCH
    APP --> MINIO
    APP --> MAILPIT
    APP --> RABBITMQ
    APP --> VAULT

    HORIZON --> PGSQL
    HORIZON --> REDIS

    PGSQL[pgsql<br/>PostgreSQL]
    MYSQL[mysql_monitoring<br/>MySQL]
    REDIS[redis<br/>Redis]
    MEMCACHED[memcached<br/>Memcached]
    MEILISEARCH[meilisearch<br/>Search]
    MINIO[minio<br/>Storage]
    MAILPIT[mailpit<br/>Email]
    RABBITMQ[rabbitmq<br/>Message Broker]
    VAULT[vault<br/>Secrets]

Volume Mounts¶

sail-pgsql              → PostgreSQL data
sail-mysql-monitoring   → MySQL monitoring data
sail-redis              → Redis data
sail-meilisearch        → Meilisearch indexes
sail-minio              → MinIO objects
sail-rabbitmq           → RabbitMQ data
sail-vault-data         → Vault secrets
sail-vault-logs         → Vault audit logs

🔄 Infrastructure as Code (IaC) Considerations¶

Docker Compose Structure¶

services:
  # Application Layer
  - laravel.test        # Main app
  - horizon             # Queue worker
  - docs                # MkDocs

  # Data Layer
  - pgsql               # Primary DB
  - mysql_monitoring    # Monitoring DB

  # Cache & Session
  - redis               # Multi-DB cache/session/queue
  - memcached           # Application cache

  # Message Broker
  - rabbitmq            # Event messaging

  # Search & Storage
  - meilisearch         # Full-text search
  - minio               # Object storage (dev)

  # Security & Dev Tools
  - vault               # Secrets management
  - mailpit             # Email testing
  - selenium            # Browser testing

Production Considerations¶

For production IaC (Terraform, Kubernetes, Docker Swarm):

1. Separate Services:
2. Database: Managed PostgreSQL (AWS RDS, DO Managed DB)
3. Cache: Managed Redis (ElastiCache, Redis Cloud)
4. Storage: Cloudflare R2 or AWS S3

5. Search: Managed Meilisearch Cloud

6. Scalability:

7. Horizontal scaling for Laravel app (load balancer)
8. Multiple Horizon workers
9. Redis Cluster for high availability

10. Separate RabbitMQ cluster

11. Security:

12. Vault in HA mode with external storage
13. TLS/SSL for all connections
14. Network isolation (VPC, subnets)

15. Secret rotation

16. Monitoring:

17. External APM (New Relic, Datadog)
18. Log aggregation (ELK, Loki)
19. Metrics (Prometheus, Grafana)

📝 Service Health Checks¶

Health Check Endpoints¶

🎨 Network Topology¶

graph TB
    subgraph "sail Network Bridge"
        APP[laravel.test]
        PGSQL[pgsql]
        MYSQL[mysql_monitoring]
        REDIS[redis]
        MEMCACHED[memcached]
        RABBITMQ[rabbitmq]
        MEILISEARCH[meilisearch]
        MINIO[minio]
        VAULT[vault]
        MAILPIT[mailpit]
        SELENIUM[selenium]
        HORIZON[horizon]
        DOCS[docs]
    end

    subgraph "Host Machine"
        HOST[localhost]
    end

    HOST -.->|:80| APP
    HOST -.->|:5173| APP
    HOST -.->|:5432| PGSQL
    HOST -.->|:3307| MYSQL
    HOST -.->|:6379| REDIS
    HOST -.->|:11211| MEMCACHED
    HOST -.->|:5672, :15672| RABBITMQ
    HOST -.->|:7700| MEILISEARCH
    HOST -.->|:9000, :8900| MINIO
    HOST -.->|:8200| VAULT
    HOST -.->|:8025| MAILPIT
    HOST -.->|:8000| DOCS

📚 Additional Resources¶

• Database Migrations Guide
• Vault Secrets Management
• Local Domain Setup
• Audio Processing Microservices

Last Updated: October 17, 2025
Version: 1.0.0
Maintained By: RawDigs DevOps Team