DStream Log Output Provider

A simple Go output provider for the DStream data streaming platform that gracefully logs any input data to stderr without JSON parsing errors.

Overview

The Log Output Provider is designed to:

• Receive data envelopes via stdin from DStream orchestration
• Handle both JSON-parseable and raw log lines gracefully
• Log all data to stderr with structured logging
• Never fail on JSON parse errors - just log the raw line
• Integrate seamlessly with the DStream ecosystem

Features

Graceful Input Handling

• Attempts to parse each line as JSON for structured logging
• Falls back to raw line logging if JSON parsing fails
• No errors or failures - all input is logged

Simple Configuration

The provider accepts a JSON command envelope:

{
  "command": "run",
  "config": {
    "logLevel": "info"
  }
}

Available Options:

• logLevel: "debug", "info", "warn", "error" (default: "info")

Building

Prerequisites

• Go 1.25+

Build Commands

# Build cross-platform binaries
make build

# Build, create manifest, and push OCI image
make push

# Clean build artifacts
make clean

# Rebuild from scratch
make rebuild

Usage

Standalone Testing

# Test with simple JSON data
echo '{"command":"run","config":{"logLevel":"info"}}
{"data":{"test":true},"metadata":{"seq":1}}' | ./plugin.darwin_arm64

# Test with log lines (no JSON)
echo '{"command":"run","config":{"logLevel":"info"}}
2025/10/24 07:25:00 [INFO] Some log line
2025/10/24 07:25:01 [ERROR] An error occurred' | ./plugin.darwin_arm64

Integration with DStream

Update dstream.hcl:

task "mssql-test" {
  type = "providers"
  
  input {
    provider_ref = "ghcr.io/katasec/dstream-ingester-mssql:v0.0.55"
    config {
      # ... MSSQL provider config
    }
  }
  
  output {
    provider_ref = "ghcr.io/katasec/dstream-log-output-provider:v0.1.0"
    config {
      logLevel = "info"
    }
  }
}

Then run:

go run . run mssql-test

Architecture

Data Flow

1. Configuration: Receives JSON command envelope via stdin (first line)
2. Data Processing: Reads JSON envelopes from stdin (subsequent lines)
3. Logging: Outputs to stderr (never stdout, no JSON failures)
4. Graceful Shutdown: Handles SIGTERM for clean exit

Communication Protocol

• stdin: Configuration envelope + data envelopes
• stdout: None (all output to stderr to avoid contamination)
• stderr: All logging and messages

OCI Distribution

Publishing

# Tag the release
git tag v0.1.0
git push origin v0.1.0

# Build and push
make push

Image Reference

ghcr.io/katasec/dstream-log-output-provider:v0.1.0

Implementation Notes

Why this provider?

• The console output provider tries to parse all stdin as JSON
• MSSQL provider produces log lines that aren't JSON
• This causes "Failed to parse JSON" errors
• Log provider fixes this by accepting any input format

Key Design Decisions:

• Logs to stderr only (never stdout)
• Attempts JSON parsing but gracefully falls back
• No validation or strict requirements
• Simple, focused responsibility (just log things)

Contributing

Feel free to submit issues or pull requests for improvements.

License

MIT License