Services: Audit, Logging & Telemetry

Architecture

TelemetryPipeline (shared orchestrator)
└── BoundedChannelQueue (backpressure-aware)
    └── BatchWriter (fan-out to sinks)

BeepLog (IBeepLog) ───────────────┐
BeepAudit (IBeepAudit) ────────────┤──→ TelemetryPipeline ──→ Sinks
Extensions (Audit/Log/Diagnostics) ┘

Pipeline stages (per envelope):
1. Enricher     → add correlation/context
2. Deduper      → suppress duplicates
3. Sampler      → probabilistic/rate-based sampling
4. Redactor     → PII/secret scrubbing
5. RateLimiter  → token bucket throttling
6. BatchWriter  → fan-out to sinks

TelemetryPipeline

Shared, opt-in telemetry orchestrator backing both BeepLog and BeepAudit:

public sealed partial class TelemetryPipeline : IAsyncDisposable
{
    private readonly BoundedChannelQueue _queue;
    private readonly BatchWriter _writer;
    private readonly IReadOnlyList<ITelemetrySink> _sinks;
    private readonly IReadOnlyList<IEnricher> _enrichers;
    private readonly IReadOnlyList<IRedactor> _redactors;
    private readonly IReadOnlyList<ISampler> _samplers;
    private readonly IMessageDeduper _deduper;
    private readonly IRateLimiter _rateLimiter;

    public event Action<string, Exception> SinkErrored;

    // Submit an envelope to the pipeline
    public ValueTask SubmitAsync(TelemetryEnvelope envelope);

    // Graceful shutdown
    public ValueTask FlushAsync(CancellationToken token);
}

Pipeline Stages

Backpressure Modes

public enum BackpressureMode { Drop, Block, Throw }

BeepAudit

Audit recorder with tamper-evidence, GDPR purge, and compliance querying:

public sealed partial class BeepAudit : IBeepAudit
{
    public BeepAudit(BeepAuditOptions options,
        TelemetryPipeline pipeline, IHashChainSigner signer = null);

    // Record an audit event
    public ValueTask RecordAsync(string action, string actor,
        string resource, Dictionary<string, object> metadata);

    // Compliance services (Phase 10)
    public void AttachComplianceServices(
        IAuditQueryEngine queryEngine = null,
        GdprPurgeService purgeService = null,
        IntegrityVerifier integrityVerifier = null);

    // Tamper-evidence (hash chain)
    public bool VerifyIntegrity(string entryId);
}

public class BeepAuditOptions
{
    public bool Enabled { get; set; } = false;
    public bool HashChain { get; set; } = false;
    public RetentionOptions Retention { get; set; }
}

Partial Files

Sub-components

BeepLog

Structured logging with severity levels, scopes, and filtering:

public sealed partial class BeepLog : IBeepLog
{
    public BeepLog(BeepLoggingOptions options, TelemetryPipeline pipeline);

    public bool IsEnabled { get; }
    public BeepLogLevel MinLevel { get; }

    // Core logging
    public void Log(BeepLogLevel level, string message,
        Dictionary<string, object> extra = null);

    // Extension methods (via BeepLogScopeExtensions)
    public void Trace(string msg);
    public void Debug(string msg);
    public void Info(string msg);
    public void Warn(string msg);
    public void Error(string msg, Exception ex = null);
    public void Critical(string msg, Exception ex = null);
}

public enum BeepLogLevel { Verbose, Debug, Information, Warning, Error, Critical }

public class BeepLoggingOptions
{
    public bool Enabled { get; set; } = false;
    public BeepLogLevel MinLevel { get; set; } = BeepLogLevel.Information;
}

Sinks & Processing

Service Extensions

Usage Examples

Setup Audit and Logging

var builder = Host.CreateApplicationBuilder();

// Bootstrap BeepDM itself
IBeepService beep = builder.Services.AddBeepServices()
    .WithDirectory(AppContext.BaseDirectory)
    .WithAppRepo("MyApp")
    .WithConfigType(BeepConfigType.Application)
    .AsSingleton()
    .Build();

// Add audit with tamper-evidence.
// NOTE: Enabled defaults to false - both pipelines are opt-in.
builder.Services.AddBeepAudit(auditOpts =>
{
    auditOpts.Enabled       = true;
    auditOpts.HashChain     = true;   // default true
    auditOpts.RetentionDays = 365;
});

// Add structured logging
builder.Services.AddBeepLogging(logOpts =>
{
    logOpts.Enabled  = true;
    logOpts.MinLevel = BeepLogLevel.Information;
});

var host = builder.Build();

Recording Audit Events

var audit = serviceProvider.GetRequiredService<IBeepAudit>();

await audit.RecordAsync("PipelineModified", "admin",
    "customer-sync",
    new Dictionary<string, object>
    {
        ["OldBatchSize"] = 100,
        ["NewBatchSize"] = 500,
        ["Timestamp"] = DateTime.UtcNow
    });

// Verify integrity
bool valid = audit.VerifyIntegrity("entry-id-123");

Structured Logging

var log = serviceProvider.GetRequiredService<IBeepLog>();

log.Info("Pipeline started");
log.Debug($"Batch size: {config.BatchSize}");

try
{
    await pipeline.ExecuteAsync();
    log.Info("Pipeline completed successfully");
}
catch (Exception ex)
{
    log.Error("Pipeline failed", ex);
    log.Critical("Pipeline critical failure - data may be inconsistent", ex);
}