Termboard

Appearance

How to Handle Catalog Schema Drift & Re-Imports ​

Target Audience: Data Stewards, Technical Mappers, Data Architects, Stakeholders
Module: Data Mapping Studio

Overview ​

External technical catalogs (Snowflake databases, OpenMetadata assets, dbt models, CSV data dictionaries) undergo frequent changes. Columns get added, moved, renamed, or deleted upstream.

When you re-import an updated catalog file or re-sync via REST API in Termboard's Data Mapping Studio, the Schema Drift Engine automatically intercepts the re-import, calculates structural changes, and presents an interactive diff preview so no verified business mappings are lost.

Re-Import Workflow ​

[ Active Mapping Session ] 
          │
          ▼  Re-import CSV / Fetch REST API
[ Structural Diff Detection Engine ]
          │
          ▼  Preview Modal
[ Review Changes: +Added, -Removed, ~Modified, ->Renamed ]
          │
          ▼  Choose Reconciliation Strategy
[ Apply Reconciliation ] ──► Session Updated & Verified Mappings Preserved

1. Triggering a Re-Import ​

In your active Data Mapping Studio workspace:

  1. Click Catalog in the top navigation bar.
  2. Select Import File... (CSV/JSON) or Fetch API (OpenMetadata).
  3. If an active mapping session exists, Termboard will not overwrite your work. Instead, the Catalog Schema Re-Import Diff modal automatically opens.

2. Reviewing Schema Differences ​

The diff preview modal groups changes into clear status tabs:

  • Added (+): Newly introduced catalog columns/attributes.
  • Removed (-): Columns present in your previous session but absent in the incoming schema.
  • Modified (~): Columns whose data type or description changed.
  • Renamed / Moved (->): Columns whose names or qualified paths changed, detected via similarity scoring.
  • Impacted Mappings: Highlights verified mappings that reference removed, modified, or renamed columns.

3. Selecting Reconciliation Options ​

At the bottom of the diff modal, configure how Termboard should handle the changes:

OptionBehaviorRecommended Setting
Keep removed items as OrphanedPreserves mapping entries for removed columns, tagging status as orphaned so audit trails and comments remain intact.ON (Checked)
Auto-migrate mappings for RenamedAutomatically updates existing mappings to point to new column names when high-confidence fuzzy matches are detected.ON (Checked)
Auto-match candidates for New ItemsLaunches background AI/similarity candidate matching on newly added columns immediately.ON (Checked)

4. Stakeholder Hands-On Test Walkthrough ​

Use the following step-by-step scenario and test files to demonstrate schema drift protection to stakeholders.

Sample Test Datasets ​

Dataset 1: Initial Catalog Schema (bcbs239_catalog_v1.csv) ​

Save the following content to a local file named bcbs239_catalog_v1.csv:

csv
container,name,description,datatype,sourcesystem
Customer,customer_id,Unique identifier for customer,VARCHAR(36),BCBS239_DW
Customer,first_name,Customer given name,VARCHAR(100),BCBS239_DW
Customer,last_name,Customer family name,VARCHAR(100),BCBS239_DW
Customer,dob,Date of birth,DATE,BCBS239_DW
Customer,email,Primary contact email address,VARCHAR(255),BCBS239_DW
Customer,phone,Primary phone number,VARCHAR(30),BCBS239_DW
Customer,fax_number,Fax contact number,VARCHAR(30),BCBS239_DW
Customer,address_line1,Street address line 1,VARCHAR(255),BCBS239_DW
Customer,city,City name,VARCHAR(100),BCBS239_DW
Customer,postal_code,Zip or postal code,VARCHAR(20),BCBS239_DW
Account,account_id,Unique account number,VARCHAR(36),BCBS239_DW
Account,customer_id,Foreign key to Customer,VARCHAR(36),BCBS239_DW
Account,account_type,Type of bank account (checking/savings),VARCHAR(50),BCBS239_DW
Account,balance,Current account ledger balance,"DECIMAL(18,2)",BCBS239_DW
Account,currency,ISO currency code,VARCHAR(3),BCBS239_DW
Account,status,Account operational status,VARCHAR(20),BCBS239_DW
Account,opened_date,Date account opened,DATE,BCBS239_DW
Transaction,tx_id,Unique transaction ID,VARCHAR(36),BCBS239_DW
Transaction,account_id,Foreign key to Account,VARCHAR(36),BCBS239_DW
Transaction,tx_amount,Transaction monetary amount,"DECIMAL(18,2)",BCBS239_DW
Transaction,tx_type,Debit or credit indicator,VARCHAR(10),BCBS239_DW
Transaction,tx_timestamp,Transaction timestamp,TIMESTAMP,BCBS239_DW
Transaction,terminal_id,POS or ATM terminal identifier,VARCHAR(50),BCBS239_DW
Loan,loan_id,Unique loan agreement ID,VARCHAR(36),BCBS239_DW
Loan,principal_amount,Original loan principal,"DECIMAL(18,2)",BCBS239_DW
Loan,interest_rate,Annual percentage rate,"DECIMAL(5,4)",BCBS239_DW
Loan,maturity_date,Loan expiration date,DATE,BCBS239_DW

Dataset 2: Drifted Catalog Schema (bcbs239_catalog_v2_drifted.csv) ​

Save the following content to a local file named bcbs239_catalog_v2_drifted.csv:

csv
container,name,description,datatype,sourcesystem
Customer,customer_id,Unique identifier for customer,VARCHAR(36),BCBS239_DW
Customer,first_name,Customer given name,VARCHAR(100),BCBS239_DW
Customer,last_name,Customer family name,VARCHAR(100),BCBS239_DW
Customer,dob,Date of birth,DATE,BCBS239_DW
Customer,email,Primary contact email address,VARCHAR(320),BCBS239_DW
Customer,primary_phone,Primary phone number,VARCHAR(30),BCBS239_DW
Customer,tax_id_number,National tax identification number,VARCHAR(50),BCBS239_DW
Customer,address_line1,Street address line 1,VARCHAR(255),BCBS239_DW
Customer,city,City name,VARCHAR(100),BCBS239_DW
Customer,postal_code,Zip or postal code,VARCHAR(20),BCBS239_DW
Account,account_id,Unique account number,VARCHAR(36),BCBS239_DW
Account,customer_id,Foreign key to Customer,VARCHAR(36),BCBS239_DW
Account,account_type,Type of bank account (checking/savings),VARCHAR(50),BCBS239_DW
Account,balance,Real-time cleared ledger balance,"DECIMAL(18,2)",BCBS239_DW
Account,credit_score,Customer credit rating score,INTEGER,BCBS239_DW
Account,currency,ISO currency code,VARCHAR(3),BCBS239_DW
Account,status,Account operational status,VARCHAR(20),BCBS239_DW
Account,opened_date,Date account opened,DATE,BCBS239_DW
Transaction,tx_id,Unique transaction ID,VARCHAR(36),BCBS239_DW
Transaction,account_id,Foreign key to Account,VARCHAR(36),BCBS239_DW
Transaction,transaction_amount,Transaction monetary amount,"DECIMAL(18,2)",BCBS239_DW
Transaction,tx_type,Debit or credit indicator,VARCHAR(10),BCBS239_DW
Transaction,tx_timestamp,Transaction timestamp,TIMESTAMP,BCBS239_DW
Transaction,ip_address,Client IP address for fraud detection,VARCHAR(45),BCBS239_DW
Loan,loan_id,Unique loan agreement ID,VARCHAR(36),BCBS239_DW
Loan,principal_amount,Original loan principal,"DECIMAL(18,2)",BCBS239_DW
Loan,interest_rate,Annual percentage rate,"DECIMAL(5,4)",BCBS239_DW
Loan,maturity_date,Loan expiration date,DATE,BCBS239_DW

Step-by-Step Test Procedure for Stakeholders ​

Step 1: Initialize Mapping Session ​

  1. Open test version of termboard: test.termboard.com
  2. Open the a model, for example: File > Open > Examples > Generic Terms
  3. Then open View > Mapping Studio.
  4. Click Catalog $\rightarrow$ Import File...
  5. Select bcbs239_catalog_v1.csv.
  6. Observe 27 catalog attributes loaded into the left LDM panel.

Step 2: Establish Verified Mappings ​

  1. Select Customer.phone and click Verify to link it to a semantic term.
  2. Select Transaction.tx_amount and click Verify to link it to a monetary term.
  3. Select Customer.fax_number and click Verify to create a baseline mapping.

Step 3: Trigger Re-Import & Structural Diff ​

  1. Click Catalog $\rightarrow$ Import File...
  2. Select bcbs239_catalog_v2_drifted.csv.
  3. Notice that your workspace is not overwritten. The Catalog Schema Re-Import Diff modal pops up automatically.

Step 4: Inspect Structural Changes ​

  1. Observe summary counts in top badges:
    • 3 Added: tax_id_number, credit_score, ip_address
    • 2 Removed: fax_number, terminal_id
    • 2 Modified: email (type expanded), balance (description updated)
    • 2 Renamed: phone $\rightarrow$ primary_phone, tx_amount $\rightarrow$ transaction_amount
  2. Click tab Renamed to review auto-detected column rename pairs.
  3. Click tab Impacted Mappings to verify that verified business mappings are flagged for review.

Step 5: Apply Reconciliation ​

  1. Keep default strategy checkboxes checked:
    • Keep removed items as Orphaned
    • Auto-migrate mappings for Renamed
    • Auto-match candidates for New Items
  2. Click Apply Reconciliation.
  3. Verify results in mapping workspace:
    • Customer.primary_phone retains the verified mapping previously linked to Customer.phone.
    • Customer.fax_number mapping status displays Orphaned with historical audit trail intact.
    • Newly added attributes display AI-generated similarity suggestions.

Summary ​

With Schema Drift protection, data stewards can continuously re-sync upstream data dictionaries without fear of losing verified glossary links or breaking audit history.