Data Warehouse
Data Warehouse
Data Warehouse
ETL and Data Model
User Guide
No part of this document may be reproduced or transmitted in any form or by any means,
electronic or mechanical, for any purpose, without the express written permission of TEMENOS Holdings NV.
Table of Contents
Overview .................................................................................................................................................. 4
The T24 Data Model and Data Warehouse Architecture ................................................................. 4
Philosophy and Standards for the Data Warehouse Data Model ...................................................... 36
Centralising all banking arrangements ........................................................................................... 36
Managing Balances ........................................................................................................................ 36
Categorisation of Tables in Data warehouse ................................................................................. 37
Maintenance of Reference Data .................................................................................................... 42
Standardisation of Segmentation across tables ............................................................................. 43
Difference between segments & hierachies within static attributes ............................................... 43
Updates & Calculations within the Data Warehouse ..................................................................... 43
Profitability ...................................................................................................................................... 44
Programming Requirements .......................................................................................................... 44
MIS reporting ..................................................................................................................................... 45
Profitability Analysis ....................................................................................................................... 45
Setup ..................................................................................................................................................... 47
jBASE Release .................................................................................................................................. 47
T24 ETL Architecture ......................................................................................................................... 47
Online Services ..................................................................................................................................... 47
Automated Extraction of Table Data .................................................................................................. 47
Selecting tables to populate the data warehouse .......................................................................... 47
Rules Routine ................................................................................................................................. 48
XML Schema generation in T24 ........................................................................................................ 51
T24 Transformation Tables ................................................................................................................ 51
EB.DDL.SCHEMA .......................................................................................................................... 51
1..1 EB.TRANSFORM ................................................................................................................ 53
T24 DW Loading Table ...................................................................................................................... 53
DW.CONNECT.PARAM ................................................................................................................. 53
T24-DW mapping mechanism ........................................................................................................... 54
Creation of DW schema ................................................................................................................. 55
Mapping using ALTOVA ................................................................................................................. 56
Virtual fields in Standard Selection ................................................................................................ 65
External Database Population ........................................................................................................... 66
Running the ETL process .................................................................................................................. 68
TSA.SERVICE record .................................................................................................................... 68
TSA.WORKLOAD.PROFILE .......................................................................................................... 68
DW API records.............................................................................................................................. 68
Overview
The Data Warehouse functionality in T24 enables the automated online extraction of data from T24
through online services. This data is mapped to fit an analysis data model, and is then loaded into an
external Data Warehouse.
Temenos Data Warehouse provides a defined Data Model, as well as the mapping required to convert
data from T24 into that Data Model. In addition to this, Temenos Data Warehouse provides a number
of enquiries and reports based on this data model, to fit a variety of banking reporting requirements.
The T24 Data Warehouse ETL (Extract, Transform, Load) functionality enables the automated
conversion of the OLTP Data in T24 into relational tables. These tables are stored on an external
separate RDBMS system. A separate driver is provided by Temenos to enable the automatic flow of
mapped data from T24 to the RDBMS.
However, it is also understood that a client may already have an existing Data Warehouse and
associated Data Model. Therefore, the mapping and enquiries provided by T24 are not mandatory but
are highly flexible; the bank may choose to specify their own mapping from T24 to fit their specific
requirements.
This document details the installation procedure for generating the T24 Data Warehouse, including the
mapping functionality for converting T24 Data into the T24 Data Model and the Loading functionality to
populate the external database.
The below section explains the methodology behind the development of the T24 data model, to assist
in the development of reports and enquiries. :
The main entites of the data model are given below:
• Party is an entity (individual or organization) which is related to the Bank (Party itself) in
someway or the other. Examples: Primary and secondary account holders (customers), loan
brokers, employees, beneficiaries, household members, counterparties to an agreement,
obligors, claimants to collateral, securitization investors, guarantors
• Arrangement is contract between two parties like customer and Bank for product or service
like collateral. Examples: Checking Account, Savings Account, Time Deposit, Retirement
Account, Collateral agreement, Limit agreement, Loan, Credit Card, Collateral Agreement,
Internal Investment Holding, Letter of Credit, Brokerage Account w/ Portfolio, Commercial
Account, Services Provider contract
• Product is any marketable offering or service including terms and conditions by the bank
which can be used by the customers. Examples of product with product characteristics: Loan,
Mortgage, Certificate of Deposit, Line of Credit, Platinum credit card, Mutual Fund, Small
Business Loan, Investment Instrument (e.g. Temenos common stock)
• Event - Any incident that happened which may involve contact with the party. Examples:
Deposit, Withdrawal, Credit/debit card charge, Complaint/Inquiry, Address Change, Balance
Inquiry, Internet page view, Brokerage buy/sell, Installment Payment, Payment Request
(claim)
• Finance – The internal accounting of the Bank’s business is described along with the financial
reporting. GL_entry and GL_Reporting _balance Examples: Management reporting,
Product profitability, Business unit profit, Arrangement Profit
• Protection Asset – It is an asset of the party and will include guarantee, credit derivatives, etc.
which may be used for credit mitigation. Examples : Automobile, real estate, jewelry,
machinery, boats, inventory, financial instruments, securities, etc
Table
Name Comment
AGMT Highest level table for each contract. This holds information
relating to all sustained contracts in the database. The table holds
information which would not normally be expected to change
through the life of the contract - however, if the contract does
change, a new record would be generated, with the new details and
a new ARRANGEMENTCODE - while the ARRANGEMENTID
would remain the same.
AGMT_AGMT_RELATIONSHIP This entity identifies relationships between arrangements and
identifies the type of relationship. An example would be the
arrangements are related for collateral purpose or one arrangement
replaced another arrangement due to a refinance. There is one
instance for each arrangement-arrangement relationship.
AGMT_BAL_POST_VALUE_BOOKING This is an off-shoot of the Agmt_Balance table for the post value
dated booking amount. The primary key of this table is the
concatenation of the '@ID + POST.VALUE.DATE+ BOOKING
DATE.
AGMT_BALANCE This file holds the various balances related to the arrangement.
Examples of balances - outstanding balance, provision balance,
interest payable/recievable balance, overdue balance/in grace
balance etc. Outstanding balance is likely to be the principal one
used. Each balance is distinguished by Balance Type, which itself
holds information about the balance itself.
There will also be a rate at which the balance is gaining
Table
Name Comment
interest/depreciating in value.
There may also be a repricing/transfer price rate applicable to this
balance. There is likely to be a counterpart BALANCE record to
the ARRANGEMENT_BALANCE for interest details.
This will hold all data relating to the financial balances of contracts.
For example, there will be information holding the amount and
currency of balances, along with information detailing the rate that
the balances are accruing interest (if applicable), or possibly
depreciating in value, and the future date that these rates may
change, the date that this balance is closed (if known), as well as
reference to the parent Arrangement contract.
Every arrangement has at least one balance - generally for the
outstanding amount. However, there may be other balances such as
interest payable/receivable, payments past due/in grace/non accrual
etc.
A maximum of one record per balance type per contract per day is
sustained in the system. This has to be stored in order to enable
historic comparisons and reporting of amounts. Therefore,
whenever there is a new balance on a new day, a new record should
be generated to reflect this.
It is debateable as to whether there should be an 'enddate' for the
balance, if one balance (but not the contract overall) is due to be
'closed' at a particular time.
AGMT_BALANCE_POST_VALUE This table has the post value dated balance of the arrangement
thereby taking care of the back valued transactions. The primary
key of this table is the concatenation of '@ID +
POST.VALUE.DATE
AGMT_BALANCE_VALUE This table is an off shoot of the Agmt_balance table having the
value dated balances of the arrangement. The primary key of this
table is concatenation of @ID + VALUE.DATE
AGMT_PARTY This associative entity defines the relationships that this
arrangement has with parties. For example, relationships include
arrangement holder, beneficiary, trustee, the car dealer that
originated the account, the business or individual that funded the
deposit money, or another customer that referred this customer to
the bank.
AGMT_PARTY_ROLE It stores the description of various arrangement party role types.
AGMT_PROFIT This entity tracks the changes to an account's profitability over
time. This entity has the account profitability that is calculated
outside of this system. There is one instance for each account, date
and profit type combination.
This entity can be derived from other entities in the model that have
costs and revenue associated with it.
AGMT_RELATIONSHIP This minor entity describes the type of relationship between two
arrangements such as collateral, transfer, etc.
BRANCH This stores the information of the branch from where the
arrangement originated. It is a type of business unit
COUNTRY It stores the details about the country along with the region.
COUNTRY_CURRENCY This stores the relationship of the country with the currency
CURRENCY It stores details about the currency
CURRENCY_CURRENCY_MARKET This stores the relation between the currency market and the
particular currency.
CURRENCY_MARKET This stores the description of the various currency markets for a
currency.
CURRENCY_RATE It stores the currency rates along with the history
EMPLOYEE_DEPT It holds the department to which an employee belongs.
EMPLOYEE_PARTY It stores the details about the employee of the organisation
EMPLOYEE_RELATION It stores the emplyee relationship with the employer
EXTERNAL_IDENTIFICATION It stores the details of external identification of a party like
passport, driving license, SSN, etc.
GENDER It stores the gender description
GL_ENTRY It stores all the GL entries over time
GL_REPORTING_BALANCE The entity that indicates the actual account balance amounts for
defined GL accounts.
GLOBAL_PARTY This stores the unique identifier of Global party and the description
Table
Name Comment
INDUSTRY It describes the industry details of the party
INTEREST_BASIS This stores the various basis of interest calculation like A/a , A/360.
etc along with the description
INTEREST_BENCHMARK It stores the benchmark rate for a particular benchmark like LIBOR
over time.
INTEREST_BENCHMARK_DES This entity represents an interest index . An example of indices are
bank prime rate, 3-month bank CD, LIBOR 6-month etc. Interest
rates charged to customers are based on these indexes.
JOB_TITLE It stores the job title of the individual.
LIMIT This table stores all the limit details of an arrangement.
LIMIT_BRANCH This table stores the limit applicable to a particular branch
LIMIT_COLLATERAL This stores the mapping of collateral with the limit.
LIMIT_COLLATERAL_PERIOD This table stores the limit collateral relationship of LIMIT with
the COLLATERAL along with the period of validity and amount of
the Collateral.
LIMIT_COLLATERAL_RIGHT It stores the relationship between the Limit and Collateral right.
LIMIT_COMMODITY It stores the limit applicable to a particular commodity
LIMIT_COUNTRY This stores the the limit for each country in the overall limit
LIMIT_CURRENCY It stores the limit applicable to a particular currency.
LIMIT_GRADE This stores the the limit for each country in the overall limit
LIMIT_MATURITY It stores the limit applicable for a period.
LIMIT_PRODUCT This stores the relationship of the Limit with various products.
MIS_DATE It stores the calendar time date, week, month, quarter, half year and
year
OCCUPATION It stores the occupation of the individual.
PARTY A party is any individual or organization that is of interest to the
Bank.
PARTY_ADDRESS It stores the mail address of the customer or party
PARTY_EMPLOYMENT This table stores the employment details of the customer.
PARTY_LIMIT It stores the total limit for the Party
PARTY_NAME It stores the name of the Party over time.
PARTY_PARTY_RELATION It stores the relationship between 2 parties with the role code
PARTY_RATING It stores the rating of the Party over time.
PARTY_RELATION Types of relationships that parties can have to each other.
Examples of relationships :
For party :
Sibling
Cohabit
Parent
For companies :
Head office
Subsidiary
Supplier
Partner
General :
Trustee
POA
Table
Name Comment
The rules for setting segmentation will be stored as part of the
extraction tool, so when a party is entered its segmentation is
identified, and this is attached to the DW table.
PARTY_STATUS This stores the status of the Party whether a defaulter , bankrupt,
etc
PRODUCT It stores the details about the Product offered by the bank
PROT_ASSET_RIGHT It stores the details about the Product offered by the bank
PROT_ASSET_RIGHT_BRANCH This stores the Collateral right for a branch
PROT_ASSET_RIGHT_LIMIT This stores the collateral right related to a limit.
PROT_ASSET_RIGHT_PARTY This stores the Collateral right for a customer
PROTECTION_ASSET This entity identifies all the known assets of Parties. These also
include equipment or vehicles being leased by the financial
institution. There is one instance for each asset. An asset could be
any item that is of value to the Party The term 'asset' is from the
customer's perspective.
PROTECTION_ASSET_GROUP This groups various collateral types into broader groups.
PROTECTION_ASSET_TYPE This stores the collateral type like securioties, physical, financial,
etc.
RATING It stores the description of a Rating of agency like S & Pwhich is a
type of Party for the Bank
RATING_AGENCY This stores the details of the particular rating agency.
REGION This stores the region details for the country
REGION_COUNTRY It stores the relation between a Country and a particular region like
Asia, Europe, EMEA, etc.
SECTOR It stores the sector description of the customer
TIME_PERIOD_TYPE It stores the description of time period type like daily, weekly,
monthly, etc.
1 : Week day
2 : Week end
3 : Holiday
Week_End_Ind CHAR(1) Indicator whether the day No
belongs to weekend
Month_End_Ind CHAR(1) Indicator whether the day No
belongs to month end
Quarter_End_Ind CHAR(1) Indicator whether the day No
belongs to quarter end
Half_Year_End_Ind CHAR(1) Indicator whether the day No
belongs to half year end
Year_End_Ind CHAR(1) Indicator whether the day No
belongs to year end
Fiscal_Year_End_Ind CHAR(1) Indicator whether the day No
belongs to fiscal year end
Time_Month_Quarter_Ind CHAR(1) A month is divided ointo 4 No
quarters. This refers to the
quarter number within the
month
Time_Total_Day_Count NUMBER The number of days till date of No
extraction
Time_Total_Week_Count NUMBER The number of weeks till date of No
extraction
Time_Total_Month_Count NUMBER The number of months till date No
of extraction
Time_Total_Year_Count NUMBER The number of years till date of No
extraction
Record_Start_Date DATE Start date of the record No
Record_End_Date DATE End date of the record No
Source_System VARCHAR2(20) Source of the extract No
Managing Balances
For any one arrangement, there can be several balances : for example, for a loan, there can be the
initial balance, the outstanding balance, the interest payable on that balance, the unpaid balance in
grace and the unpaid overdue balances. For limits, there can be initial amount, outstanding amount,
utilised amount.
For portfolios, we can have balances of shares or bonds. Instead of a financial amount, we would
store a nominal amount, with reference to the share this amount relates to.
Each balance reflects a specific point in time. As the data warehouse is engineered to enable
complete and accurate historic reporting, we will sustain all balances as they change across time.
Therefore, if a balance changes on a different day, the previous balance will not be overwritten but a
new balance record will be generated. If however a balance is changed twice in one day, the more
recent balance will overwrite the previous balance as it is only necessary after COB to store a single
balance snapshot per day.
Entity
Entity
Name Type Definition
AGMT Fact Highest level table for each contract. This
holds information relating to all sustained
contracts in the database. The table holds
information which would not normally be
expected to change through the life of the
contract - however, if the contract does
change, a new record would be generated,
with the new details and a new
ARRANGEMENTCODE - while the
ARRANGEMENTID would remain the
same.
AGMT_AGMT_RELATIONSHIP Dimension This entity identifies relationships between
arrangements and identifies the type of
relationship. An example would be the
arrangements are related for collateral
purpose or one arrangement replaced
another arrangement due to a refinance.
There is one instance for each arrangement-
arrangement relationship.
AGMT_BAL_BOOK_DATE Fact
AGMT_BAL_POST_VALUE_BOOKING Fact
AGMT_BALANCE Fact This file holds the various balances related
to the arrangement.
Examples of balances - outstanding balance,
provision balance, interest
payable/recievable balance, overdue
balance/in grace balance etc. Outstanding
balance is likely to be the principal one
used. Each balance is distinguished by
Balance Type, which itself holds
information about the balance itself.
There will also be a rate at which the
balance is gaining interest/depreciating in
value.
There may also be a repricing/transfer price
rate applicable to this balance. There is
likely to be a counterpart BALANCE record
to the ARRANGEMENT_BALANCE for
interest details.
This will hold all data relating to the
financial balances of contracts. For
example, there will be information holding
the amount and currency of balances, along
with information detailing the rate that the
balances are accruing interest (if
applicable), or possibly depreciating in
value, and the future date that these rates
may change, the date that this balance is
closed (if known), as well as reference to
the parent Arrangement contract.
Every arrangement has at least one balance -
generally for the outstanding amount.
Entity
Name Type Definition
However, there may be other balances such
as interest payable/receivable, payments
past due/in grace/non accrual etc.
A maximum of one record per balance type
per contract per day is sustained in the
system. This has to be stored in order to
enable historic comparisons and reporting of
amounts. Therefore, whenever there is a
new balance on a new day, a new record
should be generated to reflect this.
It is debateable as to whether there should
be an 'enddate' for the balance, if one
balance (but not the contract overall) is due
to be 'closed' at a particular time.
AGMT_BALANCE_POST_VALUE Fact
AGMT_BALANCE_VALUE Fact
AGMT_PARTY Dimension This associative entity defines the
relationships that this arrangement has with
parties. For example, relationships include
arrangement holder, beneficiary, trustee, the
car dealer that originated the account, the
business or individual that funded the
deposit money, or another customer that
referred this customer to the bank.
AGMT_PARTY_ROLE Dimension It stores the description of various
arrangement party role types.
AGMT_PROFIT Fact This entity tracks the changes to an
account's profitability over time. This entity
has the account profitability that is
calculated outside of this system. There is
one instance for each account, date and
profit type combination.
This entity can be derived from other
entities in the model that have costs and
revenue associated with it.
AGMT_RELATIONSHIP Dimension This minor entity describes the type of
relationship between two arrangements such
as collateral, transfer, etc.
BRANCH Dimension This stores the information of the branch
from where the arrangement originated. It is
a type of business unit
COUNTRY Dimension It stores the details about the country along
with the region.
COUNTRY_CURRENCY Dimension This stores the relationship of the country
with the currency
CURRENCY Dimension It stores details about the currency
CURRENCY_CURRENCY_MARKET Dimension This stores the relation between the
currency market and the particular currency.
CURRENCY_MARKET Dimension This stores the description of the various
currency markets for a currency.
CURRENCY_RATE Fact It stores the currency rates along with the
history
EMPLOYEE_DEPT Dimension It holds the department to which an
employee belongs.
EMPLOYEE_PARTY Dimension It stores the details about the employee of
the organisation
EMPLOYEE_RELATION Dimension It stores the emplyee relationship with the
employer
EXTERNAL_IDENTIFICATION Dimension It stores the details of external identification
of a party like passport, driving license,
SSN, etc.
GENDER Dimension It stores the gender description
Entity
Name Type Definition
GLOBAL_PARTY Dimension This stores the unique identifier of Global
party and the description
INDUSTRY Dimension It describes the industry details of the party
INTEREST_BASIS This stores the various basis of interest
Dimension calculation like A/a , A/360. etc along with
the description
INTEREST_BENCHMARK Fact It stores the benchmark rate for a particular
benchmark like LIBOR over time.
INTEREST_BENCHMARK_DES Dimension This entity represents an interest index . An
example of indices are bank prime rate, 3-
month bank CD, LIBOR 6-month etc.
Interest rates charged to customers are
based on these indexes.
JOB_TITLE Dimension It stores the job title of the individual.
LIMIT Fact This table stores all the limit details of an
arrangement.
LIMIT_BRANCH Fact
LIMIT_COLLATERAL Fact This stores the mapping of collateral with
the limit.
LIMIT_COLLATERAL_PERIOD Fact
LIMIT_COLLATERAL_RIGHT Fact
LIMIT_COMMODITY Fact
LIMIT_COUNTRY Fact This stores the the limit for each country in
the overall limit
LIMIT_CURRENCY Fact
LIMIT_GRADE Fact This stores the the limit for each country in
the overall limit
LIMIT_MATURITY Fact
For party :
Sibling
Cohabit
Parent
For companies :
Head office
Subsidiary
Supplier
Partner
General :
Entity
Name Type Definition
Trustee
POA
REGION Dimension This stores the region details for the country
REGION_COUNTRY Dimension It stores the relation between a Country and
a particular region like Asia, Europe,
EMEA, etc.
SECTOR Dimension It stores the sector description of the
customer
TIME_PERIOD_TYPE Dimension It stores the description of time period type
like daily, weekly, monthly, etc.
The data model of the Party and its related tables are shown above.
In order to group tables into equal levels of granularity, we use the concept of segmentation.
Examples of segmentation are:
• Party Segmentation: grouping parties into different ‘types’.
• Product Segmentation : Grouping products
For example, we could define parties into the following segmentation groups:
• Young Unmarried Low earner
• Young Unmarried High Earner
• Young Married Low Earner
• Young Married High Earner etc etc.
This segmentation is clearly based on 3 attributes (age bracket, marital status, income bracket). Once
the segments have been defined, they can be defined in all applicable tables: so the PL Balances
table, which may be consolidated by party segment, would use these segmentation rules. In addition,
the Party table would also use these rules to provide standard grouping for parties,
One important item to note is that segmentation should not cross tables – product segmentation
relates purely to data relevant to that product, party segmentation purely to details about the party.
An additional note, which may be for phase 2 but which would be extremely useful for regulatory
reporting, is the development of parallel segments. In this way, a party may fit into one type of
segment for one method of reporting (Young, Unmarried Low Earner) and an entirely different
segment for a different method of reporting (Individual, I.T Graduate, Inner London). Obviously, when
items are consolidated by segment there would need to be parallel balances tables to handle each
segment, or the consolidation would need to occur at a yet lower level with references to the segments
which are applicable.
collateral amounts, when based on T24 contracts: these will still be extracted from T24, even if they
could be derived from the DW data.
Of course, it is possible that some balances may be influenced by contracts outside of T24. In this
case, additional processing would be required – however, we are currently building a self-reliant T24
Data Warehouse, local configuration when combining with other systems is inevitable.
Profitability
Profitability calculations present some of the more complicated and important challenges to the data
model.
In T24, each Profit/Loss Posting has its own entry in the CATEG.ENTRY file. However, it is
anticipated that were these postings to be migrated across to the data warehouse, the potential
volume of transactions would cause prohibitive performance issues. Therefore, it is proposed that all
PL postings from the OLTP (Online Transaction Processor – in most cases, T24) are consolidated into
PL Balances files.
Consolidation can be based on any variables within the profitability sphere. However, it is anticipated
that the main sources of consolidation will be :
Party or Party Segment (at various levels of party segmentation)
Arrangement, or Arrangement Segmentation
Entry Type, or Entry Type Segmentation
Daily, or through Date Segmentation (periods).
The level of consolidation for these postings can be within the control of the client. It may be that
consolidation is placed at the party and arrangement level, totalling by individual PL type and stored
daily : this would produce the highest volume of balances, stored at the lowest level. This would be
appropriate for a bank with relatively low volumes of transactions.
However, a higher volume retail client may wish to total all PL entries by a higher level of party,
arrangement and pl segmentation and for these entries to be stored on a weekly or even monthly
basis. This would produce a lower volume of balances.
A further possibility is that the bank may choose to consolidate some entries at the lowest level, and
others at a higher level (e.g all high net worth clients would have pl entries consolidated by customer,
whereas low net worth clients would have entries consolidated by various entry levels).
Profitability will then be calculated by grouping balances, making adjustments from the pl adjustments
table and taking into account activites from the events file.
Programming Requirements
To ensure that the DW solution is platform-independent, and also to make maximum use of our
existing expertise (i.e. in T24 and Infobasic) wherever possible, the maximum amount of work has
been performed in T24, and the minimal amount within the data store. However, even with this in mind,
some calculations may be performed on the data warehouse.
MIS reporting
Profitability Analysis
Phase 1 of DW also delivers Profitability Analysis, the first of a set of business intelligence applications
– others will follow in later stages.
DW will provide equivalent reporting functionality to the existing T24 Management Information module,
through reports generated from the external Data Store. The current Management Information module
provides information including income and expense, average balances, transaction counts, etcetera
about customers, products, departments from which Management may analyse profitability and
performance.
Elements within the MI module regarded as efficient will be reused, while those areas that require
improvement will be replaced with external best-practice software. Overall, the analytical potential of
the new environment will considerably exceed that possible today with the Management Information
module. The figure below illustrates the functionality currently delivered by MI, and how this
functionality will now be replicated through DW.
MI Element DW Equivalent
PL Reallocation Entries P&L Adjustments can be calculated online using features of the
ETL product, along with manual adjustments using
MI.AUTO.ENTRY entries – so MI.AUTO.ENTRY is generated as
before, and exported to the DW, for both Transfer Pricing entries
and PL Reallocations. However, this functionality will need to be
further enhanced to run online / through services rather than only
at the building of the MI cube as initially delivered.
MI Data Cubes T24 Functionality is not used for the generation of data cubes.
Data Cubes are generated using the OLAP reporting tool.
With data stored on an external data store, it will be possible to provide all of the reports currently
delivered by the MI module. In addition, performance issues related to the generation of MI Data
Cubes and Balance Movement records will be removed. The process will be more transparent to
Users, who will be able to take greater ownership of the MI process.
Setup
The Data Warehouse T24 Code and configured tables are supplied to T24 in the same way as all
other modules, through addition of the DW product code to the SPF through the standard installation
procedures. As with all T24 modules, once the functionality has been installed, all unauthorised
records relating to DW need to be authorised.
jBASE Release
As the DW Functionality makes use of new jBASE functionality to enable the automatic population of
alternative tables and databases, the minimal release upon which the installation of DW can be done
is 4.1.5.11.
Online Services
Automated Extraction of Table Data
The data to be extracted from T24 to the data warehouse is controlled via the DW.EXTRACT.PARAM
table.
The DW.EXTRACT.PARAM table specifies which T24 tables will have some or all of their data
extracted into the data warehouse
Shown below is a typical DW.EXTRACT.PARAM setup for the T24 application Customer with Rules
routine:
Rules Routine
This is a standard EB.API routine to which the whole of extracted record will be passed in VM/FM format along
with the @ID. The routine can decide based on any of the field’s value to determine whether each record is of
interest to Data Warehouse or not.
If the routine decides not to proceed with the extract then it needs to send a return value of 1 in the fourth
argument.
The field Rules.Routine is a non-mandatory field. If a routine is attached then above said processing will happen
else the record will be extracted from T24 unconditionally.
We could write an Enquiry on this table and export the Schema’s for mapping purposes.
1..1 EB.TRANSFORM
This table holds the generic mapping XSL for mapping from T24 table to a DW table.
@ID syntax: <Valid T24 Table name>-<UPDATE/DELETE>. If the XSL contains the mapping logic for
the table update (input/edit) then the ID should be <Table name>-UPDATE. If it contains the mapping
logic for delete, then the ID should be <Table name>-DELETE. The field “Mapping.xsl” stores the
mapping XSL for Update/Delete. It is left to the user to input a valid and meaningful XSL.
The existing T24 data needs to be mapped in a relational format so that it can be stored in a relational
database which is currently Oracle. Mapping the T24 data is achieved by the following means:
• XML schema of T24 table (source table) to be mapped is taken from Standard Selection
• The destination table data model or DDL is produced in ERWIN and the corresponding XML
schema is produced using EB.DDL.SCHEMA in T24. It should be kept in mind that while
producing DDL, the data type and length needs be done after considering the data dictionary
in Standard Selection in T24.
• Thereafter both the schemas are inserted in Altova Mapforce as shown below and the
mapping is done manually using Mapforce features
• In case of multivalues, the mulitvalue fields are populated in another table with the Primary
key being a concatenation of the Parent table Primary Key in T24 and the first multivalue field.
Similarly a sub value fields are populated in another table with the Primary key being a
concatenation of the Parent table Primary Key in T24 , the first multivalue field and the first
field in the sub-value
• In case of concatenation a ‘*’ is used to differentiate the various concatenated fields.
• After physical mapping in Mapforce, the XSL or the stylesheet of the mapping is generated
using the XSLT tab in Mapforce.
• The stylesheet is subsequently integrated with other stylesheets, if required and is pasted in
EB.TRANSFORM of T24. This style sheet is referred by the T24 XML record to produce the
corresponding DW XML record which is subsequently turned to a SQL record using
DW.TRANSFORM
Creation of DW schema
The user needs to go to EB.DDL.SCHEMA and type the name of the table which one wants
to map in DW (data warehouse) and the press Edit tab.
The User needs to type the short description and description of the DW table schema and then copy
the corresponding DDL of the DW table into DDL and commit the deal. The user needs to verify the
deal subsequently.
The DW schema subsequently produced needs to be copied in notepad and stored in a file.
Mapping the source and destination tables by dragging the cursor and using Altova features
The stylesheet is produced by pressing the XSLT tab in Altova as shown below:
Step 2: Click the Preview button on any one of the tables (i.e. based on the order of the DW tables in
which the INSERT/UPDATE needs happen)
Step 3: Now, view the XSL by clicking the XSLT tab at the bottom of the Mapping window (make sure
that the Preview button of the DW table is Pressed as shown)
Step 4: On the XSLT Tab, right-click and click Select All and then Copy to copy the XSL
Step 5: Paste the copied XSL in an Editor (notepad or preferably a XML editor)
• We could remove any unwanted comment or/and the XSD references.
• Make sure the Slash (/) before the element value “row” is removed.
We could validate this XSL for correctness, if we are using a XML Editor.
The same could be repeated for mapping containing more two DW tables and a single Integrated XSL
could be produced.
I Descriptor field.
Subroutine
Subroutine
JDescriptor field.
Under WINDOWS:
TSA.SERVICE record
The TSA Service record DW.EXTRACT is released from the core. This service record gets released
either as <XYZ>/DW.EXTRACT or DW.EXTRACT depending upon whether the set-up is Multi-
company or not (where XYZ represents the company mnemonic). Before authorizing the record, the
SERVICE.CONTROL field needs to be changed as AUTO and it needs to be linked to a valid TSA
Workload profile. Also the corresponding F.BATCH record (DW.EXTRACT or <XYZ>/DW.EXTRACT)
should be authorised.
TSA.WORKLOAD.PROFILE
TSA.WORKLOAD.PROFILE record for this task must be created with as many agents as required,
and linked to the Service record DW.EXTRACT. For more details on setting up TSA services please
refer to the Technical User Guide.
DW API records
The T24 DW module has three extension points i.e. ETL.
• Firstly, the extract routine that is called once the data extract has been performed (E).
• The second is used to transform the contents to suitable for loading into the Data
Warehouse(T)
• Third, to determine the database connectivity parameters, and the actually load the data into
an external database(L)
In the above-said ETL, the Loading is implemented via Java API.
DW.BULK.EXTRACT
This is the table in which we specify the T24 table(s) from which records need to be extracted and
populated in an external data store.
TSA.SERVICE record
The TSA Service record DW.BULK.PROCESSING is released from the core. This service record gets
released either as <XYZ>/DW.BULK.PROCESSING or DW.BULK.PROCESSING depending upon
whether the set-up is Multi-company or not (where XYZ represents the company mnemonic). Before
authorizing the record, the SERVICE.CONTROL field needs to be changed as AUTO and it needs to
be linked to a valid TSA Workload profile. Also the corresponding F.BATCH record
(DW.BULK.PROCESSING or <XYZ>/ DW.BULK.PROCESSING) should be authorised.
TSA.WORKLOAD.PROFILE
TSA.WORKLOAD.PROFILE record for this task must be created with as many agents as required,
and linked to the Service record DW.BULK.PROCESSING. For more details on setting up TSA
services please refer to the Technical User Guide
Note: For DW Bulk Extract/Upload the Service <XYZ>/DW.EXTRACT should continue to run in
addition to <XYZ>/DW.BULK.PROCESSING.
The selection fields, “Fld.Selection”, “Selection.Op.” and “Selection.Cr” are optional. (Figure1)
The “Extract.Reg” field is also optional and an alternative to the selection fields.
It accepts only a valid REG record as illustrated in Figures 2 and 3.
If both the selection fields and the REG are used, the selection fields serve as pre-select criteria.
Figure.1
Figure.2
Figure.3
Figure.5
DW.EXTRACT:
Figure.6
DW.BULK.EXTRACT: After the services DW.BULK.PROCESSING and DW.EXTRACT are
started;
Figure.7
Error Messages
ETL error logging facility
The existing T24 DW ETL system is enhanced to log the errors in the ETL stage. There
could be various errors happening in these three layers of Extraction, Transformation &
Loading and it becomes essential to log the errors as when it occurs. These error logs could
be manually analyzed/reconciled, errors corrected and ETL process could be re-tried. The
errors are logged with sufficient information like Date, Time, Error Stage, Routine name,
Application, Module, File Name, Record Id and the most important the Error message
EB.LOGGING.PARAMETER
This is the parameter file for the log. This parameter will be keyed in by Context.id and will contain two
fields called LOG.LEVEL & LOG.METHOD for DW.EXTRACT, DW.TRANSFORM and DW.LOAD.
LOG LEVEL
1. Info: Includes INFO (information about the ETL), ERROR and FATAL errors.
2. Error: Includes ERROR and FATAL errors.
3. Fatal: Includes only FATAL error.
LOG METHOD
1. Always: Logs all messages even when the ETL crashes via FATAL .ERROR.
2. Console: Prints the output in the service and does not log in EB.LOGGING.
3. Disk: All logs are written to EB.LOGGING under normal circumstances. Does not log when
the ETL crashes via the T24 subroutine FATAL.ERROR.
ENQUIRY DW.RECONCILE.PARAM
A new enquiry has been written on F.EB.LOGGING, which takes in the select criteria’s and produces
a report like Enquiry on the error log.
The output is split date-wise and contains the count of INFO, ERROR and FATAL logs.
Interfaces
Altova Mapforce Mapping Guide
Please visit the below link for using Altova Mapforce for mapping T24 table(s) to DW (Data
Warehouse) table(s).
www.altova.com/download/2006/MapForcePro.pdf
Building a table
The user needs to click the entity icon to build a table with ERWIN in the logical mode as shown in the
figure below and name the table along with its definition by right clicking the table icon and selecting
‘Entity Properties’ . Thereafter the user needs to type the attributes for the table and define the data
type, primary key and definition of the attributes in the table after choosing the ‘Attributes’ .
Definition of the Primary key of the Party Table by clicking the ‘Primary key’
One can create an identifying relationship when you add a relationship using the Relationships dialog,
on the diagram window by clicking the Identifying relationship icon. AllFusion ERwin DM automatically
migrates primary key attributes from a parent entity to a child entity, so you do not need to enter any
foreign keys in the child.
Tip: You can click a relationship line to highlight it. Highlighting allows you to trace the path of the
relationship line from parent to child.
a) crows feet with a cross and a circle, as shown below (Zero, one or more)
b) crows feet with a cross (One or more)
In a mandatory non-identifying relationship, the attributes that are migrated into the non-key area of
the child entity are required in the child entity. Therefore, the foreign key cannot be null.
One can set the default null option in the Model Properties dialog. If you assign a non-identifying
relationship, AllFusion ERwin DM automatically assigns either a mandatory or optional non-identifying
relationship based on your default null setting. However, you can change the null option in the
Relationships dialog to override the default setting.
AllFusion ERwin DM draws a mandatory non-identifying relationship differently depending on the
notation for your model:
IE notation draws a dashed line with a cross on the parent end. The child end indicates the
cardinality with:
a) crows feet with a cross and a circle, as shown below (Zero, one or more)
b ) crows feet with a cross (One or more)
One can change the null option in the Relationships dialog to override the default setting.
c) Click ‘Preview’ tab for the generation of the DDL as shown below:
Refer to the ERWIN user guide for further details on using ERWIN for data modelling.
The user thereafter needs to paste this DDL in the database like Oracle as shown below and execute
it to create the relevant table in database.
Glossary
DW Data Warehouse/Data Warehousing
ETL Extract, Transform and Load
XSL Extensible Stylesheet Language
XSLT Extensible Stylesheet Language Transformations
XML Schema(XSD) XML Schema Definition
XML Extensible Markup Language
PL Profit and loss
MI Management Information
GL General Ledger
DDL Data Definition Language