Technical Folder Guides

Module-level documentation for DataManagementEngineStandard internals

Accuracy note (2026)

The engine uses partial-class and helper-manager patterns across multiple modules (e.g., Editor/BeepSync, Editor/Importing, ConfigUtil). These guides map the canonical module ownership.

Table of Contents

Core Modules

ModulePathPurpose
Root DataManagementEngineStandard/ Main project with editor, services, helpers, and utilities
ConfigUtil ConfigUtil/ Configuration management with ConfigEditor facade and specialized managers
ConfigUtil.Managers ConfigUtil/Managers/ DataConnectionManager, QueryManager, EntityMappingManager, ComponentConfigManager, MigrationHistoryManager
Editor Editor/ Core editor classes: DMEEditor, ETL, UOW, Forms, Sync, Importing, Mapping, Defaults
Helpers Helpers/ Data source helpers, type mapping, RDBMS helpers, universal helpers
WebAPI WebAPI/ WebAPIDataSource and helpers for REST API connectivity

Editor Subsystem

The Editor directory contains the core data management functionality:

Editor/DM/

  • DMEEditor.cs - Main orchestrator (IDMEEditor implementation)
  • DMEEditor.DataSource.cs - Data source lifecycle management
  • DMEEditor.Config.cs - Configuration access

Editor/ETL/

  • ETL_BK/ - Legacy ETL runtime (ETLEditor, ETLDataCopier, etc.)
  • Engine/ - Modern pipeline engine (PipelineEngine, PipelineManager)
  • Engine/Workflow/ - Workflow engine integration
  • Registry/ - PipelinePluginRegistry
  • Scheduling/ - SchedulerHost, PipelineRunQueue, WatermarkTracker
  • Observability/ - MetricsEngine, AlertingEngine, ObservabilityStore

Editor/UOW/

  • UnitofWork.Core.cs - Core UnitOfWork implementation
  • UnitofWork.CRUD.cs - CRUD operations
  • UnitofWork.Core.Extensions.cs - Delete, commit, rollback, events
  • UnitofWork.Core.Utilities.cs - Navigation, tracking, filtering
  • UnitofWork.OBLIntegration.cs - ObservableBindingList integration
  • UnitOfWorkFactory.cs - Runtime type creation
  • UnitOfWorkWrapper.cs - Late-bound access wrapper

Editor/Forms/

  • FormsManager.Core.cs - Main coordinator (30+ partial files)
  • Helpers/ - RelationshipManager, DirtyStateManager, EventManager
  • Configuration/ - Performance and behavior settings
  • Models/ - Form state and data models

Editor/BeepSync/

  • BeepSyncManager.Core.cs - Sync orchestrator
  • Helpers/ - Sync helpers and utilities
  • Interfaces/ - Sync contracts

Editor/Importing/

  • DataImportManager.cs - Import orchestrator
  • Helpers/ - Validation, transformation, batch, progress helpers
  • Interfaces/ - Import contracts

Editor/Mapping/

  • MappingManager.cs - Field mapping (8 partial files)
  • Core/ - Core mapping logic
  • Configuration/ - Mapping settings
  • Helpers/ - Mapping utilities

Editor/Defaults/

  • DefaultsManager.cs - Default value management
  • Interfaces/ - Default resolver contracts
  • Resolvers/ - Built-in default value resolvers
  • Helpers/ - Default value utilities

Helpers & Providers

RDBMSHelpers

  • RDBMSHelper.cs - Main RDBMS helper facade
  • RDBMSDataTypeHelper.cs - Data type mapping
  • RDBMSSchemaHelper.cs - Schema operations

UniversalDataSourceHelpers

HelperPurpose
Core/Base helper functionality
RdbmsHelpers/SQL generation, query building
MongoDBHelpers/MongoDB-specific operations
RestApiHelpers/REST API query building
VectorHelpers/Vector database operations
GraphHelpers/Graph database operations
SearchHelpers/Search engine integration
StreamingHelpers/Stream processing
TimeSeriesHelpers/Time-series database operations
RedisHelpers/Redis operations
CassandraHelpers/Cassandra operations
FileHelpers/File-based data source operations
Analysis/Data analysis utilities
Mapping/Data type mapping

Caching & JSON

Caching

  • CacheManager.cs - Central cache coordinator
  • Providers/ - SimpleCacheProvider, MemoryCacheProvider, RedisCacheProvider, HybridCacheProvider
  • DataSources/ - CachedMemoryDataSource, InMemoryCacheDataSource
  • Examples/ - Usage examples

JSON

  • JsonDataSource.cs - Basic JSON datasource
  • JsonDataSourceAdvanced.cs - Advanced with helpers
  • JsonExtensions.cs - JToken/JObject extensions
  • Helpers/ - Schema, filter, CRUD, async, graph, cache helpers

Pipeline & ETL

Modern pipeline-based ETL engine:

  • Editor/ETL/Engine/PipelineEngine.cs - Core execution engine
  • Editor/ETL/Engine/PipelineManager.cs - Persistence and orchestration
  • Editor/ETL/Engine/PipelineStepRunner.cs - Step execution
  • Editor/ETL/Engine/PipelineCheckpointManager.cs - Fault tolerance
  • Editor/ETL/Engine/PipelineLineageTracker.cs - Data lineage
  • Editor/ETL/Registry/PipelinePluginRegistry.cs - Plugin discovery
  • Editor/ETL/Scheduling/SchedulerHost.cs - Cron scheduling
  • Editor/ETL/Observability/MetricsEngine.cs - KPI computation
  • Editor/ETL/Observability/AlertingEngine.cs - Rule-based alerting

Workflow

Multi-step business process execution:

  • DataManagementModelsStandard/Workflow/WorkFlow.cs - Workflow definition
  • DataManagementModelsStandard/Workflow/WorkFlowStep.cs - Individual step
  • DataManagementModelsStandard/Workflow/WorkFlowAction.cs - Executable action
  • DataManagementModelsStandard/Workflow/WorkFlowTrigger.cs - Event triggers
  • DataManagementModelsStandard/Workflow/WorkFlowRetryPolicy.cs - Retry configuration
  • Editor/WorkFlowEditor.cs - Design-time editor

Rules Engine

Business rules and validation:

  • Rules/RulesEngine.cs - Main engine
  • Rules/RulesParser.cs - Rule text parser
  • Rules/RuleRegistry.cs - Rule registration
  • Rules/BuiltinRules/ - Built-in rule categories (Compute, DQ, Date, Enrich, Flow, etc.)
  • Rules/BuiltinParsers/ - NfelParser, SqlWhereParser, JsonRuleTreeParser, etc.
Previous: Advanced Tools Next: API Reference