Generally Accepted Data Management Principles

(GADMP)
Susan Earley
Agenda
• What is a Principle?
• What makes a good Principle?
• Management Principle vs. Design Principle
• Other Generally Accepted Principles
• Proposed: Generally Accepted Data Management Principles
 What is a Principle?
• A Principle is a statement that represents
what is desirable and positive, used to
determine rightfulness or wrongfulness of
an action. Principles are more basic than
policy and objectives, and are meant to
govern both.
(paraphrased from http://www.businessdictionary.com/definition/principles.html)

• Principles are rules or laws that are

permanent, unchanging, and universal in
nature.
(The 7 Habits of Highly Effective People, Stephen R. Covey)
 What makes a Good Principle?
• Universally Applicable
• Applies to all parts of the system, all use cases, and all situations
• Easily Understood
• Simply defined, not complex or complicated
• Uses common language
• Always True
• Actions that align with the principle always deliver positive results.
• Actions that disregard the principle always deliver undesirable results.
 Management Principle vs. Design Principle
Management Management Principles Design Principles Design Principles (Water
Principles (Water Example) Example)
States the WHAT Public Availability States the HOW Water pressure minimums
General Oversight Structurally Sound Specific Implementation Options Pipe materials to use
Generic Health Preservation Tailored/Customized Water testing system integration
Universally Continuous Improvement Not universally applicable to the Reservoir Maintenance
Applicable organization

Data Sources
and Input

Scores,
Forecasts
Operational
Systems

Data
Data Lake Library External Data
Targets
Other Generally Accepted Principles
GAAP (Accounting) GARP (Record Keeping) GACAP (Commodity Accountability)
• Regularity • Accountability • Organization
• Consistency • Transparency • Financial Accountability
• Sincerity • Integrity • Internal Controls and Audit
• Permanence of Methods • Protection • Commodity Inventory Management
• Non-Compensation • Compliance • Compliance
• Prudence • Availability
• Continuity • Retention http://pdf.usaid.gov/pdf_docs/P
• Periodicity • Disposition nadd540.pdf
• Materiality
• Utmost Good Faith

GASSP (System Security)

GAPTL (Principles for Teaching and Learning)
Gartner's GAIP (Information)
Other Principles
Information Architecture Principles, DDHIDS
(paraphrased) GEO DMP (Earth Observations)
• Reflection of Business, not Technology • Metadata for Discovery
• Business Involvement in Definition • Online Access
• Enterprise Perspective for Data Management • Data Encoding
• Shared Definitions by Consensus • Data Documentation
• Reusability Drives Value • Data Traceability
• Data Management Staff Value • Data Quality Control
• One Size Does Not Fit All • Data Preservation
• Conceptual and Physical Models Must Tie • Data and Metadata Verification
• Data Management Provides Value to Business • Data Review and Reprocessing
• Formal and Proactive > Improvised and Reactive • Persistent and Resolvable Identifiers
http://dbhids.org/wp- https://www.earthobservations.org/documents/geo_xii/GEO-
content/uploads/2017/09/OCIO_DBHIDS-Data-Governance- XII_10_Data%20Management%20Principles%20Implementation
Framework-Strategic-Plan-v2.pdf %20Guidelines.pdf
Proposed:
Generally Accepted Data Management Principles
• Worth as an Asset • Operational Incorporation
• Ethical Use • Operational Transparency
• Cultural Incorporation • Prevention > Remediation
• Data Lifecycle Ownership • Reuse Improves Value
Principle 1: Worth as an Asset
Data is an asset, with measurable
tangible value and intangible business
value.
• Measurement of Tangible Value
• Base Value: Market price to buy equivalent data
• Revenue: Generated by Opportunities Discovered
• Monetization: Generated by Sale of Data / Data Services
• Risk: Fines and Fees due to Data Loss
• Expense: Cost of Data Storage and Maintenance
• Measurement of Intangible Business Value
• Reputation Improvement: Public perception of
organization regarding proper data handling and customer
data protection
• Reputation Loss: Data Loss impact on Revenues
• Opportunity Projections: Revenue from Opportunities
• Opportunity Costs: Revenue Missed due to lack of data
• Best Practice: Calculate Tangible Value and Intangible
Impact Value as line items on balance sheets.
Principle 2: Ethical Use
Data must only be used for Ethical purposes.
• Each organization defines ‘Ethical Use’ details subject
to applicable laws and regulations.
• Data use must be audited regularly to ensure violations
are caught promptly and actions are taken to prevent
future violations.
• Best Practice: Ensure data access is granted for approved
purposes, logged for audit, and re-validated regularly.
• Best Practice: Ensure data access is revoked promptly when roles
change or users leave.
Principle 3: Cultural Incorporation
Data Management must be a
company-wide program to ensure
positive results.
• Everyone in an organization is responsible
for implementing and complying with Data
Management requirements.
• Best Practice: Integrate Data Management
processes with Change Management processes
• Best Practice: Create dedicated teams to develop
training & communication materials and facilitate
implementation.
• Best Practice: Create incentives tied to compliance
and data quality ratings.
Principle 4: Data Lifecycle Ownership
Data Lifecycle Ownership is the shared oversight responsibility
for what happens to data within your environment.
• Data lifecycle owners make decisions that affect
• the actual data (internally generated or copies of externally sourced data) stored
throughout its lifecycle - acquisition, storage, protection, usage, archiving, and
destruction
• the metadata, including definitions, validation rules and quality levels,
transformation rules, access requirements and restrictions, sharing rules and
restrictions, and usage rules and restrictions
• Candidates for Data Lifecycle Owners are those who have a need for the data as
well as understand the data. Best candidates are those with both.
• Best Practice: Clearly define data lifecycle owners by function, point of entry, or subject
area. Review assignments occasionally to prevent ‘orphans’.
• Best Practice: Data Lifecycle Owners should meet regularly as a committee to ensure
cross-communication and issue discussions.
• Best Practice: Do NOT assign by default data lifecycle ownership to technical support or
application support teams. Typically, IT department support teams do NOT understand
the data, they just follow instructions.
Principle 5: Operational Incorporation
Data Management requirements must be
understood, acceptable, achievable,
implemented pervasively, and measurable.
• Understandable, controlled, and well-managed
data and policies create positive business results.
• Absent, misunderstood, uncontrolled, or badly-
managed data and policies have negative business
and legal consequences.
• Best Practice: Create training programs and maintain
frequent communication paths with all stakeholders to
ensure understanding.
• Best Practice: Use standard tool functionality to implement
automated Data Management processes.
• Best Practice: Conduct periodic implementation
evaluations, resolve discrepancies, and publish results.
Principle 6: Operational Transparency
All data lifecycle details must be clearly and
consistently documented, published, and
auditable.
• Data lifecycle details include
• Where and how data is introduced into the environment
• How data elements are defined and named
• Where and how data is stored
• How, why, and when data is processed or transformed
• How, why, and when data is archived or purged
• How data-related decisions and controls are implemented
• Data-related policies, decision, and standards must be
publicly available for audit review.
• Best Practice: Implement a public Metadata Repository
and/or Data Catalog, and audit regularly to ensure the
metadata matches the actual data.
• data repository lists, data element definitions, and data
structures and models
• data flows, data lineage and provenance, and detailed data
mappings
Principle 7: Prevention > Remediation
Data Management starts at the point
of entry.
• Prevent unethical use or unauthorized
access rather than of remediating after-
the-fact.
• Prevent low quality or bad data from
entering your environment rather than
fixing it downstream.
• Best Practice: Implement security at every
point of data access.
• Best Practice: Validate data at the point of
entry based on requirements via
constraints, pick lists, and validation APIs.
Report violations as issues to the source
systems or providers.
Principle 8: Reuse Improves Value
Sharing data is better than making
copies.
• Data value increases with sharing.
• Isolated datasets have limited value.
• Redundant datasets have negative value.
• Best Practice: Integrate once and
provide data in a public Data Library
(with appropriate security, of course).
Discussion!