SSIS mojomo ETL-Framework

Inspired by the BI Monkey SSIS ETL Framework, this solution expands it's core concepts to significant enhancements and new aspects to solve current and upcoming challenges in the etl enviroment.

ETL-Framework

The ETL-framework is responsible for execution, monitoring and managing of the etl-process. 

Highlights:

  • Parallel processing of packages (number of threads can be configured)
  • Fully Configurable by defining metadata 
  • Recoverability
  • Logging 
  • Configuration-Overriding (during execution)
  • Execution Orders and Staging-Levels to support package dependencies
  • Full compatible with SSIS Catalog 

Bundled packages:

Follwing functionality will be supported by the bundled packages:

  • Retrieve and storing table row counts
  • Archiving automatically table data based on their lifecycle 
  • Delete automatically table data based on their lifecylce 

These packages can be customized by modifying the metadata database.

Design Patterns

Additionally design patterns (best practice packages) will be included. They can be seamlessly integrated in the etl framework. 

They include solutions for the  following requirements:

  • Full table Extraction
  • Partial table extraction by audit-column
  • Partial table extraction by database trigger
  • Partial table extraction by MSSQL CDC
  • Load table by replacing tables
  • Load SCD Type 1 
  • Load SCD Type 2
  • Load fact table 

Highlights: 

  • Runnable  (based on AdventureWorks SampleDB) 
  • Independent development 
  • Easy to use - Most Design Patterns can be adapted by setting variables
  • Standardized Parameter-Interface - ETL-Framework can override the connection parameters  

Requirements

  • SSIS 2012
  • SQL Server database

If you have any questions/feedback, you can contact me (skynet18@hotmail.com).

Last edited Mar 20, 2013 at 8:05 AM by mojomo, version 6