[go: up one dir, main page]

Hemminki, 2021 - Google Patents

Large scale data ingestion pipelines for energy industry using AWS

Hemminki, 2021

View PDF
Document ID
17971008228550690977
Author
Hemminki A
Publication year

External Links

Snippet

Data is becoming increasingly important to industries. Industries can't just use data to be more profitable, but they rather depend on it. This Master's Thesis answers questions on what are the characteristics of industrial IoT data, and how the data ingestion can be done …
Continue reading at lutpub.lut.fi (PDF) (other versions)

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30557—Details of integrating or interfacing systems involving at least one database management system
    • G06F17/30569—Details of data format conversion from or to a database
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30575—Replication, distribution or synchronisation of data between databases or within a distributed database; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30312—Storage and indexing structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for programme control, e.g. control unit
    • G06F9/06—Arrangements for programme control, e.g. control unit using stored programme, i.e. using internal store of processing equipment to receive and retain programme
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogramme communication; Intertask communication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Error detection; Error correction; Monitoring responding to the occurence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2097—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements maintaining the standby controller/processing unit updated
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06Q—DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management, e.g. organising, planning, scheduling or allocating time, human or machine resources; Enterprise planning; Organisational models
    • G06Q10/063—Operations research or analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06Q—DATA PROCESSING SYSTEMS OR METHODS, SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL, SUPERVISORY OR FORECASTING PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation, e.g. computer aided management of electronic mail or groupware; Time management, e.g. calendars, reminders, meetings or time accounting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network-specific arrangements or communication protocols supporting networked applications
    • H04L67/10—Network-specific arrangements or communication protocols supporting networked applications in which an application is distributed across nodes in the network

Similar Documents

Publication Publication Date Title
CN108628929B (en) Method and apparatus for intelligent archiving and analysis
CN107908672B (en) Application report realization method, device and storage medium based on Hadoop platform
US12367423B2 (en) System and method for ensuring that the results of machine learning models can be audited
AU2013221760B2 (en) Providing configurable workflow capabilities
WO2012047215A1 (en) Method and system for processing events
CN118377768A (en) Data ETL method, device, equipment and medium based on service flow
US20250148289A1 (en) System and method of llm training for network state conversational analytics
Martinviita Time series database in Industrial IoT and its testing tool
Khoo et al. Constraints on future analysis metadata systems in High Energy Physics
Hemminki Large scale data ingestion pipelines for energy industry using AWS
Racka Apache NIFI as a tool for stream processing of measurement data
Suleykin et al. Implementing big data processing workflows using open source technologies
Gamero et al. Scalability testing approach for Internet of Things for manufacturing SQL and NoSQL database latency and throughput
Hu et al. Automated monitoring method for enterprise microservices network operation status based on database knowledge graph
Gorton et al. Gridoptics (tm) a novel software framework for integrating power grid data storage, management and analysis
US10348596B1 (en) Data integrity monitoring for a usage analysis system
Zhao et al. A high throughput distributed log stream processing system for network security analysis
Bui Real-time Data Analytic on Google Cloud: A Complete Data Pipeline from Self-Host Databases to GCP Services
CN120386780B (en) Device data storage Doris method based on industrial Internet
US12405805B2 (en) Systems and methods for processing incident data through a data pipeline
US20250110791A1 (en) Systems and methods for applying a complex data processor and router by implementing dynamic configuration settings
US20250138971A1 (en) Systems and methods for aggregating and generating a single incident profile
CN115914379B (en) Data exchange device and data exchange system
US20250138972A1 (en) Systems and methods for aggregating and generating a daily incident profile
US20250138970A1 (en) Systems and methods for aggregating and mapping incident characteristics into daily incident profiling