From c6b08ddb93a3376544d03e467bbf89a9afc14286 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Thu, 26 May 2022 15:12:50 +0200
Subject: [PATCH 01/20] Create dependabot.yml

---
 .github/workflows/dependabot.yml | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 .github/workflows/dependabot.yml

diff --git a/.github/workflows/dependabot.yml b/.github/workflows/dependabot.yml
new file mode 100644
index 0000000..313f059
--- /dev/null
+++ b/.github/workflows/dependabot.yml
@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: "docker"
+    directory: "/"
+    schedule:
+      interval: "daily"

From e614a476a7a2642330dd4fc5a3a28c1ee4870fdf Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Tue, 31 May 2022 19:36:07 +0200
Subject: [PATCH 02/20] feat: fixed title for houyly trends

---
 index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/index.html b/index.html
index bc8413a..c5b69e1 100644
--- a/index.html
+++ b/index.html
@@ -86,7 +86,7 @@
     <div class="row">
         <div class="col-md-5">
             <legend>
-                Usage Trends <span class="accounting-cost-model"></span> - 7 days
+                Usage Trends (% per namespace) - 7 days
                 <select id="selected-usage-trend-type" onchange="FrontendApi.showSelectedUsageTrendType()">
                     <option value="default" disabled selected style="display:none;">Hourly Usage</option>
                     <option value="hourly-usage">Hourly Usage</option>

From b56a7016e4b85cee88b9d993185139e958872003 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Fri, 3 Jun 2022 08:25:58 +0200
Subject: [PATCH 03/20] chore: add codeql badge in readme

---
 README.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/README.md b/README.md
index 24410c9..6cf85de 100644
--- a/README.md
+++ b/README.md
@@ -2,6 +2,7 @@
 
 ![Apache License](https://img.shields.io/github/license/rchakode/kube-opex-analytics.svg?label=License&style=for-the-badge)
 [![GitHub Actions Status](<https://img.shields.io/github/workflow/status/rchakode/kube-opex-analytics/Build?logo=GitHub&style=for-the-badge>)](https://github.com/rchakode/kube-opex-analytics/actions)
+[![GitHub Actions Status](<https://img.shields.io/github/workflow/status/rchakode/kube-opex-analytics/CodeQL?label=CodeQL&logo=GitHub&style=for-the-badge>)]([https://github.com/rchakode/kube-opex-analytics/actions](https://github.com/rchakode/kube-opex-analytics/actions/workflows/codeql-analysis.yml))
 [![Calendar Versioning](https://img.shields.io/badge/calver-YY.MM.MICRO-bb8fce.svg?style=for-the-badge)](http://calver.org)
 ![Docker pulls](https://img.shields.io/docker/pulls/rchakode/kube-opex-analytics.svg?label=Docker%20Pulls&style=for-the-badge)
 

From 81360db1887b20a4dafcb477a8f53edbcd1985cd Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Wed, 8 Jun 2022 20:30:32 +0200
Subject: [PATCH 04/20] chore: removed codeql badge in README

---
 README.md | 1 -
 1 file changed, 1 deletion(-)

diff --git a/README.md b/README.md
index 6cf85de..24410c9 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,6 @@
 
 ![Apache License](https://img.shields.io/github/license/rchakode/kube-opex-analytics.svg?label=License&style=for-the-badge)
 [![GitHub Actions Status](<https://img.shields.io/github/workflow/status/rchakode/kube-opex-analytics/Build?logo=GitHub&style=for-the-badge>)](https://github.com/rchakode/kube-opex-analytics/actions)
-[![GitHub Actions Status](<https://img.shields.io/github/workflow/status/rchakode/kube-opex-analytics/CodeQL?label=CodeQL&logo=GitHub&style=for-the-badge>)]([https://github.com/rchakode/kube-opex-analytics/actions](https://github.com/rchakode/kube-opex-analytics/actions/workflows/codeql-analysis.yml))
 [![Calendar Versioning](https://img.shields.io/badge/calver-YY.MM.MICRO-bb8fce.svg?style=for-the-badge)](http://calver.org)
 ![Docker pulls](https://img.shields.io/docker/pulls/rchakode/kube-opex-analytics.svg?label=Docker%20Pulls&style=for-the-badge)
 

From ae0e36825cefa48aa7aeec890a89ab48130c0b1e Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:30:05 -0300
Subject: [PATCH 05/20] Minor devel/debug experience changes

---
 .gitignore   |  3 ++-
 backend.py   |  9 +++++----
 run-debug.sh | 15 +++++++++------
 3 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/.gitignore b/.gitignore
index cf17744..5b9124d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,4 +2,5 @@
 .idea/
 .tox/
 venv/
-__pycache__/
\ No newline at end of file
+__pycache__/
+*.s[q-z]p
diff --git a/backend.py b/backend.py
index 705ef22..6e37569 100644
--- a/backend.py
+++ b/backend.py
@@ -21,7 +21,7 @@
 import threading
 import time
 import traceback
-import urllib
+import urllib, urllib3
 from typing import Any, List
 
 import flask
@@ -38,6 +38,7 @@
 
 import werkzeug.middleware.dispatcher as wsgi
 
+urllib3.disable_warnings()
 
 def create_directory_if_not_exists(path):
     try:
@@ -716,9 +717,9 @@ def pull_k8s(api_context):
     headers = {}
     client_cert = None
     endpoint_info = urllib.parse.urlparse(KOA_CONFIG.k8s_api_endpoint)
-    if endpoint_info.hostname != '127.0.0.1' and endpoint_info.hostname != 'localhost':
+    if KOA_CONFIG.enable_debug or (endpoint_info.hostname != '127.0.0.1' and endpoint_info.hostname != 'localhost'):
         if KOA_CONFIG.k8s_auth_token != 'NO_ENV_AUTH_TOKEN':
-            headers['Authorization'] = ('%s %s' % KOA_CONFIG.k8s_auth_token_type, KOA_CONFIG.k8s_auth_token)
+            headers['Authorization'] = '%s %s' % (KOA_CONFIG.k8s_auth_token_type, KOA_CONFIG.k8s_auth_token)
         elif KOA_CONFIG.k8s_rbac_auth_token != 'NO_ENV_TOKEN_FILE':
             headers['Authorization'] = ('Bearer %s' % KOA_CONFIG.k8s_rbac_auth_token)
         elif KOA_CONFIG.k8s_auth_username != 'NO_ENV_AUTH_USERNAME' and \
@@ -748,7 +749,7 @@ def pull_k8s(api_context):
 def create_metrics_puller():
     try:
         while True:
-            KOA_LOGGER.debug('{puller] collecting new samples')
+            KOA_LOGGER.debug('[puller] collecting new samples')
 
             KOA_CONFIG.load_rbac_auth_token()
 
diff --git a/run-debug.sh b/run-debug.sh
index 8ba34c3..b62247f 100755
--- a/run-debug.sh
+++ b/run-debug.sh
@@ -14,18 +14,21 @@ if [ "$KOA_K8S_API_ENDPOINT" == "" ]; then
     exit 1
 fi
 
-KOA_K8S_AUTH_TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='kube-opex-analytics')].data.token}"|base64 -d)
+KOA_K8S_AUTH_TOKEN=$(kubectl get secrets -n kube-opex-analytics -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='kube-opex-analytics')].data.token}"|base64 -d)
 if [ "$KOA_K8S_AUTH_TOKEN" == "" ]; then
     echo "cannot find KOA_K8S_AUTH_TOKEN"
     exit 1
 fi
 
+export KOA_DB_LOCATION=${KOA_DB_LOCATION:-$PWD/db}
 export KOA_K8S_API_ENDPOINT
-export KOA_K8S_AUTH_TOKEN=
+export KOA_K8S_AUTH_TOKEN
 export KOA_ENABLE_DEBUG=true
 export KOA_K8S_API_VERIFY_SSL=false
-export KOA_BILLING_HOURLY_RATE=9.92
-export KOA_BILLING_CURRENCY_SYMBOL='$'
-export KOA_COST_MODEL='CHARGE_BACK'
+export KOA_BILLING_HOURLY_RATE=${KOA_BILLING_HOURLY_RATE:-9.92}
+export KOA_BILLING_CURRENCY_SYMBOL=${KOA_BILLING_CURRENCY_SYMBOL:-'$'}
+export KOA_COST_MODEL=${KOA_COST_MODEL:-CUMULATIVE_RATIO}
+export KOA_POLLING_INTERVAL_SEC=${KOA_POLLING_INTERVAL_SEC:-300}
+
 fuser -k 5483/tcp || true
-./entrypoint.sh
\ No newline at end of file
+./entrypoint.sh

From 0771eb50f2d46e5ddeaa2656eb5f2f61271d412e Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:31:30 -0300
Subject: [PATCH 06/20] Add options to include/exclude namespaces

---
 backend.py                     | 25 +++++++++++++++++++++++++
 docs/configuration-settings.md |  4 +++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/backend.py b/backend.py
index 6e37569..9f4e4e6 100644
--- a/backend.py
+++ b/backend.py
@@ -14,6 +14,7 @@
 import collections
 import enum
 import errno
+import fnmatch
 import json
 import logging
 import os
@@ -69,6 +70,8 @@ class Config:
     k8s_ssl_cacert = os.getenv('KOA_K8S_CACERT', None)
     k8s_ssl_client_cert = os.getenv('KOA_K8S_AUTH_CLIENT_CERT', 'NO_ENV_CLIENT_CERT')
     k8s_ssl_client_cert_key = os.getenv('KOA_K8S_AUTH_CLIENT_CERT_KEY', 'NO_ENV_CLIENT_CERT_CERT')
+    included_namespaces = [ i for i in os.getenv('KOA_INCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
+    excluded_namespaces = [ i for i in os.getenv('KOA_EXCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
 
     def __init__(self):
         self.load_rbac_auth_token()
@@ -98,6 +101,19 @@ def __init__(self):
         else:
             self.koa_verify_ssl_option = self.k8s_verify_ssl
 
+    @staticmethod
+    def match(items, pattern):
+        return any([ fnmatch.fnmatch(i, pattern) for i in items ])
+
+    @staticmethod
+    def allow_namespace(namespace):
+        if KOA_CONFIG.match(KOA_CONFIG.excluded_namespaces, namespace):
+            return False
+
+        return len(KOA_CONFIG.included_namespaces) == 0 or \
+                '*' in KOA_CONFIG.included_namespaces or \
+                KOA_CONFIG.match(KOA_CONFIG.included_namespaces, namespace)
+
     def load_rbac_auth_token(self):
         try:
             with open('/var/run/secrets/kubernetes.io/serviceaccount/token', 'r') as rbac_token_file:
@@ -336,9 +352,13 @@ def extract_namespaces_and_initialize_usage(self, data):
         for _, item in enumerate(data_json['items']):
             metadata = item.get('metadata', None)
             if metadata is not None:
+                if not KOA_CONFIG.allow_namespace(metadata.get('name')):
+                    continue
                 self.usageByNamespace[metadata.get('name')] = ResourceCapacities(cpu=0.0, mem=0.0)
                 self.requestByNamespace[metadata.get('name')] = ResourceCapacities(cpu=0.0, mem=0.0)
 
+        KOA_LOGGER.debug("Found namespaces: %s", ', '.join(self.usageByNamespace.keys()))
+
     def extract_nodes(self, data):
         # exit if not valid data
         if data is None:
@@ -407,6 +427,9 @@ def extract_pods(self, data):
         # process likely valid data
         data_json = json.loads(data)
         for _, item in enumerate(data_json['items']):
+            if not KOA_CONFIG.allow_namespace(item['metadata']['namespace']):
+                continue
+
             pod = Pod()
             pod.namespace = item['metadata']['namespace']
             pod.name = '%s.%s' % (item['metadata']['name'], pod.namespace)
@@ -453,6 +476,8 @@ def extract_pod_metrics(self, data):
         # process likely valid data
         data_json = json.loads(data)
         for _, item in enumerate(data_json['items']):
+            if not KOA_CONFIG.allow_namespace(item['metadata']['namespace']):
+                continue
             pod_name = '%s.%s' % (item['metadata']['name'], item['metadata']['namespace'])
             pod = self.pods.get(pod_name, None)
             if pod is not None:
diff --git a/docs/configuration-settings.md b/docs/configuration-settings.md
index db53c72..6331428 100644
--- a/docs/configuration-settings.md
+++ b/docs/configuration-settings.md
@@ -13,4 +13,6 @@ This section provides an exhaustive list of configuration variables used by `kub
 * `KOA_K8S_AUTH_PASSWORD` sets the password for Basic Authentication.
 * `KOA_COST_MODEL` (version >= `0.2.0`): sets the model of cost allocation to use. Possible values are: _CUMULATIVE_RATIO_ (default) indicates to compute cost as cumulative resource usage for each period of time (daily, monthly); _CHARGE_BACK_ calculates cost based on a given cluster hourly rate (see `KOA_BILLING_HOURLY_RATE`); _RATIO_ indicates to compute cost as a normalized percentage of resource usage during each period of time.
 * `KOA_BILLING_HOURLY_RATE` (required if cost model is _CHARGE_BACK_): defines a positive floating number corresponding to an estimated hourly rate for the Kubernetes cluster. For example if your cluster cost is $5,000 dollars a month (i.e. `~30*24` hours), its estimated hourly cost would be `6.95 = 5000/(30*24)`.
-* `KOA_BILLING_CURRENCY_SYMBOL` (optional, default is '`$`'): sets a currency string to use to annotate costs on reports.
\ No newline at end of file
+* `KOA_BILLING_CURRENCY_SYMBOL` (optional, default is '`$`'): sets a currency string to use to annotate costs on reports.
+* `KOA_INCLUDED_NAMESPACES` (optional, default is '`*`'): comma-separated list of namespaces to monitore. If any value is provided, only these namespaces will be accounted.
+* `KOA_EXCLUDED_NAMESPACES` (optional, default is ''): comma-separated list of namespaces to ignore. If any value is provided, these namespaces will be discarded. This options takes precedence over `KOA_INCLUDED_NAMESPACES`.

From d81c1a0eb96dc36b21565f1c060a67c704a413f1 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:32:20 -0300
Subject: [PATCH 07/20] Dump resources.requests histogram analytics

---
 backend.py | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/backend.py b/backend.py
index 9f4e4e6..4629e3d 100644
--- a/backend.py
+++ b/backend.py
@@ -176,6 +176,15 @@ class RrdPeriod(enum.IntEnum):
                                                        ['namespace', 'resource'])
 }
 
+PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS = {
+    RrdPeriod.PERIOD_14_DAYS_SEC: prometheus_client.Gauge('koa_namespace_daily_requests',
+                                                          'Current daily resource reservation per namespace',
+                                                          ['namespace', 'resource']),
+    RrdPeriod.PERIOD_YEAR_SEC: prometheus_client.Gauge('koa_namespace_monthly_requests',
+                                                       'Current monthly resource reservation per namespace',
+                                                       ['namespace', 'resource'])
+}
+
 # create Flask application
 app = flask.Flask(__name__, static_url_path=KOA_CONFIG.static_content_location, template_folder='.')
 cors = CORS(app, resources={r"/dataset/*": {"origins": "127.0.0.1"}})
@@ -459,6 +468,7 @@ def extract_pods(self, data):
                 pod.nodeName = item['spec']['nodeName']
                 pod.cpuRequest = 0.0
                 pod.memRequest = 0.0
+                #TODO: extract initContainers
                 for _, container in enumerate(item.get('spec').get('containers')):
                     resources = container.get('resources', None)
                     if resources is not None:
@@ -696,9 +706,16 @@ def dump_histogram_analytics(dbfiles, period):
         usage_export = collections.defaultdict(list)
         usage_per_type_date = {}
         sum_usage_per_type_date = {}
+        requests_export = collections.defaultdict(list)
+        requests_per_type_date = {}
+        sum_requests_per_type_date = {}
         for _, db in enumerate(dbfiles):
             rrd = Rrd(db_files_location=KOA_CONFIG.db_location, dbname=db)
             current_periodic_usage = rrd.dump_histogram_data(period=period)
+
+            rrd = Rrd(db_files_location=KOA_CONFIG.db_location, dbname=KOA_CONFIG.usage_efficiency_db(db))
+            current_periodic_rf = rrd.dump_histogram_data(period=period)
+
             for res in [ResUsageType.CPU, ResUsageType.MEMORY]:
                 for date_key, usage_value in current_periodic_usage[res].items():
                     if usage_value > 0.0:
@@ -708,11 +725,26 @@ def dump_histogram_analytics(dbfiles, period):
                             usage_per_type_date[res][date_key] = {}
                         usage_per_type_date[res][date_key][db] = usage_value
 
+                        if res not in requests_per_type_date:
+                            requests_per_type_date[res] = collections.defaultdict(lambda: 0.0)
+                        if date_key not in requests_per_type_date[res]:
+                            requests_per_type_date[res][date_key] = {}
+
+                        rf = current_periodic_rf[res][date_key]
+                        if rf > 0.0:
+                            requests_per_type_date[res][date_key][db] = usage_value / rf
+                        else:
+                            requests_per_type_date[res][date_key][db] = 0.0
+
                         if db != KOA_CONFIG.db_billing_hourly_rate:
                             if res not in sum_usage_per_type_date:
                                 sum_usage_per_type_date[res] = collections.defaultdict(lambda: 0.0)
                             sum_usage_per_type_date[res][date_key] += usage_value
 
+                            if res not in sum_requests_per_type_date:
+                                sum_requests_per_type_date[res] = collections.defaultdict(lambda: 0.0)
+                            sum_requests_per_type_date[res][date_key] += usage_value
+
         for res, usage_data_bundle in usage_per_type_date.items():
             for date_key, db_usage_item in usage_data_bundle.items():
                 for db, usage_value in db_usage_item.items():
@@ -730,10 +762,28 @@ def dump_histogram_analytics(dbfiles, period):
                             PROMETHEUS_PERIODIC_USAGE_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
                                 usage_cost)
 
+                        requests_value = requests_per_type_date[res][date_key][db]
+                        requests_cost = round(requests_value, KOA_CONFIG.db_round_decimals)
+                        if KOA_CONFIG.cost_model == 'RATIO' or KOA_CONFIG.cost_model == 'CHARGE_BACK':
+                            requests_ratio = requests_value / sum_requests_per_type_date[res][date_key]
+                            requests_cost = round(100 * requests_ratio, KOA_CONFIG.db_round_decimals)
+                            if KOA_CONFIG.cost_model == 'CHARGE_BACK':
+                                requests_cost = round(
+                                    requests_ratio * usage_per_type_date[res][date_key][KOA_CONFIG.db_billing_hourly_rate],
+                                    KOA_CONFIG.db_round_decimals)
+                        requests_export[res].append('{"stack":"%s","requests":%f,"date":"%s"}' % (db, requests_cost, date_key))
+                        if Rrd.get_date_group(now_gmtime, period) == date_key:
+                            PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
+                                requests_cost)
+
         with open(str('%s/cpu_usage_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
             fd.write('[' + ','.join(usage_export[0]) + ']')
         with open(str('%s/memory_usage_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
             fd.write('[' + ','.join(usage_export[1]) + ']')
+        with open(str('%s/cpu_requests_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
+            fd.write('[' + ','.join(requests_export[0]) + ']')
+        with open(str('%s/memory_requests_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
+            fd.write('[' + ','.join(requests_export[1]) + ']')
 
 
 def pull_k8s(api_context):

From c75f6a1ceff404456bfa57d1ff639f47a3d89028 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Sun, 12 Jun 2022 22:50:32 +0200
Subject: [PATCH 08/20] ci: trigger build and lint on commit and pr

---
 .github/workflows/pythonpackage.yml | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml
index fb38a5e..fb895af 100644
--- a/.github/workflows/pythonpackage.yml
+++ b/.github/workflows/pythonpackage.yml
@@ -1,6 +1,12 @@
 name: Build
 
-on: [push]
+on:
+  push:
+    branches:
+    - '**'
+  pull_request:
+    branches:
+    - main
 
 jobs:
   build:

From dc15e13b91d74d90c010c2dc2bc27033524c1e76 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:30:05 -0300
Subject: [PATCH 09/20] Minor devel/debug experience changes

---
 .gitignore   |  3 ++-
 backend.py   |  9 +++++----
 run-debug.sh | 15 +++++++++------
 3 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/.gitignore b/.gitignore
index cf17744..5b9124d 100644
--- a/.gitignore
+++ b/.gitignore
@@ -2,4 +2,5 @@
 .idea/
 .tox/
 venv/
-__pycache__/
\ No newline at end of file
+__pycache__/
+*.s[q-z]p
diff --git a/backend.py b/backend.py
index 705ef22..6e37569 100644
--- a/backend.py
+++ b/backend.py
@@ -21,7 +21,7 @@
 import threading
 import time
 import traceback
-import urllib
+import urllib, urllib3
 from typing import Any, List
 
 import flask
@@ -38,6 +38,7 @@
 
 import werkzeug.middleware.dispatcher as wsgi
 
+urllib3.disable_warnings()
 
 def create_directory_if_not_exists(path):
     try:
@@ -716,9 +717,9 @@ def pull_k8s(api_context):
     headers = {}
     client_cert = None
     endpoint_info = urllib.parse.urlparse(KOA_CONFIG.k8s_api_endpoint)
-    if endpoint_info.hostname != '127.0.0.1' and endpoint_info.hostname != 'localhost':
+    if KOA_CONFIG.enable_debug or (endpoint_info.hostname != '127.0.0.1' and endpoint_info.hostname != 'localhost'):
         if KOA_CONFIG.k8s_auth_token != 'NO_ENV_AUTH_TOKEN':
-            headers['Authorization'] = ('%s %s' % KOA_CONFIG.k8s_auth_token_type, KOA_CONFIG.k8s_auth_token)
+            headers['Authorization'] = '%s %s' % (KOA_CONFIG.k8s_auth_token_type, KOA_CONFIG.k8s_auth_token)
         elif KOA_CONFIG.k8s_rbac_auth_token != 'NO_ENV_TOKEN_FILE':
             headers['Authorization'] = ('Bearer %s' % KOA_CONFIG.k8s_rbac_auth_token)
         elif KOA_CONFIG.k8s_auth_username != 'NO_ENV_AUTH_USERNAME' and \
@@ -748,7 +749,7 @@ def pull_k8s(api_context):
 def create_metrics_puller():
     try:
         while True:
-            KOA_LOGGER.debug('{puller] collecting new samples')
+            KOA_LOGGER.debug('[puller] collecting new samples')
 
             KOA_CONFIG.load_rbac_auth_token()
 
diff --git a/run-debug.sh b/run-debug.sh
index 8ba34c3..b62247f 100755
--- a/run-debug.sh
+++ b/run-debug.sh
@@ -14,18 +14,21 @@ if [ "$KOA_K8S_API_ENDPOINT" == "" ]; then
     exit 1
 fi
 
-KOA_K8S_AUTH_TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='kube-opex-analytics')].data.token}"|base64 -d)
+KOA_K8S_AUTH_TOKEN=$(kubectl get secrets -n kube-opex-analytics -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='kube-opex-analytics')].data.token}"|base64 -d)
 if [ "$KOA_K8S_AUTH_TOKEN" == "" ]; then
     echo "cannot find KOA_K8S_AUTH_TOKEN"
     exit 1
 fi
 
+export KOA_DB_LOCATION=${KOA_DB_LOCATION:-$PWD/db}
 export KOA_K8S_API_ENDPOINT
-export KOA_K8S_AUTH_TOKEN=
+export KOA_K8S_AUTH_TOKEN
 export KOA_ENABLE_DEBUG=true
 export KOA_K8S_API_VERIFY_SSL=false
-export KOA_BILLING_HOURLY_RATE=9.92
-export KOA_BILLING_CURRENCY_SYMBOL='$'
-export KOA_COST_MODEL='CHARGE_BACK'
+export KOA_BILLING_HOURLY_RATE=${KOA_BILLING_HOURLY_RATE:-9.92}
+export KOA_BILLING_CURRENCY_SYMBOL=${KOA_BILLING_CURRENCY_SYMBOL:-'$'}
+export KOA_COST_MODEL=${KOA_COST_MODEL:-CUMULATIVE_RATIO}
+export KOA_POLLING_INTERVAL_SEC=${KOA_POLLING_INTERVAL_SEC:-300}
+
 fuser -k 5483/tcp || true
-./entrypoint.sh
\ No newline at end of file
+./entrypoint.sh

From 2ee8b4a235769f75c5c4d8f7147575b0d6cf5cf5 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:31:30 -0300
Subject: [PATCH 10/20] Add options to include/exclude namespaces

---
 backend.py                     | 25 +++++++++++++++++++++++++
 docs/configuration-settings.md |  4 +++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/backend.py b/backend.py
index 6e37569..9f4e4e6 100644
--- a/backend.py
+++ b/backend.py
@@ -14,6 +14,7 @@
 import collections
 import enum
 import errno
+import fnmatch
 import json
 import logging
 import os
@@ -69,6 +70,8 @@ class Config:
     k8s_ssl_cacert = os.getenv('KOA_K8S_CACERT', None)
     k8s_ssl_client_cert = os.getenv('KOA_K8S_AUTH_CLIENT_CERT', 'NO_ENV_CLIENT_CERT')
     k8s_ssl_client_cert_key = os.getenv('KOA_K8S_AUTH_CLIENT_CERT_KEY', 'NO_ENV_CLIENT_CERT_CERT')
+    included_namespaces = [ i for i in os.getenv('KOA_INCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
+    excluded_namespaces = [ i for i in os.getenv('KOA_EXCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
 
     def __init__(self):
         self.load_rbac_auth_token()
@@ -98,6 +101,19 @@ def __init__(self):
         else:
             self.koa_verify_ssl_option = self.k8s_verify_ssl
 
+    @staticmethod
+    def match(items, pattern):
+        return any([ fnmatch.fnmatch(i, pattern) for i in items ])
+
+    @staticmethod
+    def allow_namespace(namespace):
+        if KOA_CONFIG.match(KOA_CONFIG.excluded_namespaces, namespace):
+            return False
+
+        return len(KOA_CONFIG.included_namespaces) == 0 or \
+                '*' in KOA_CONFIG.included_namespaces or \
+                KOA_CONFIG.match(KOA_CONFIG.included_namespaces, namespace)
+
     def load_rbac_auth_token(self):
         try:
             with open('/var/run/secrets/kubernetes.io/serviceaccount/token', 'r') as rbac_token_file:
@@ -336,9 +352,13 @@ def extract_namespaces_and_initialize_usage(self, data):
         for _, item in enumerate(data_json['items']):
             metadata = item.get('metadata', None)
             if metadata is not None:
+                if not KOA_CONFIG.allow_namespace(metadata.get('name')):
+                    continue
                 self.usageByNamespace[metadata.get('name')] = ResourceCapacities(cpu=0.0, mem=0.0)
                 self.requestByNamespace[metadata.get('name')] = ResourceCapacities(cpu=0.0, mem=0.0)
 
+        KOA_LOGGER.debug("Found namespaces: %s", ', '.join(self.usageByNamespace.keys()))
+
     def extract_nodes(self, data):
         # exit if not valid data
         if data is None:
@@ -407,6 +427,9 @@ def extract_pods(self, data):
         # process likely valid data
         data_json = json.loads(data)
         for _, item in enumerate(data_json['items']):
+            if not KOA_CONFIG.allow_namespace(item['metadata']['namespace']):
+                continue
+
             pod = Pod()
             pod.namespace = item['metadata']['namespace']
             pod.name = '%s.%s' % (item['metadata']['name'], pod.namespace)
@@ -453,6 +476,8 @@ def extract_pod_metrics(self, data):
         # process likely valid data
         data_json = json.loads(data)
         for _, item in enumerate(data_json['items']):
+            if not KOA_CONFIG.allow_namespace(item['metadata']['namespace']):
+                continue
             pod_name = '%s.%s' % (item['metadata']['name'], item['metadata']['namespace'])
             pod = self.pods.get(pod_name, None)
             if pod is not None:
diff --git a/docs/configuration-settings.md b/docs/configuration-settings.md
index db53c72..6331428 100644
--- a/docs/configuration-settings.md
+++ b/docs/configuration-settings.md
@@ -13,4 +13,6 @@ This section provides an exhaustive list of configuration variables used by `kub
 * `KOA_K8S_AUTH_PASSWORD` sets the password for Basic Authentication.
 * `KOA_COST_MODEL` (version >= `0.2.0`): sets the model of cost allocation to use. Possible values are: _CUMULATIVE_RATIO_ (default) indicates to compute cost as cumulative resource usage for each period of time (daily, monthly); _CHARGE_BACK_ calculates cost based on a given cluster hourly rate (see `KOA_BILLING_HOURLY_RATE`); _RATIO_ indicates to compute cost as a normalized percentage of resource usage during each period of time.
 * `KOA_BILLING_HOURLY_RATE` (required if cost model is _CHARGE_BACK_): defines a positive floating number corresponding to an estimated hourly rate for the Kubernetes cluster. For example if your cluster cost is $5,000 dollars a month (i.e. `~30*24` hours), its estimated hourly cost would be `6.95 = 5000/(30*24)`.
-* `KOA_BILLING_CURRENCY_SYMBOL` (optional, default is '`$`'): sets a currency string to use to annotate costs on reports.
\ No newline at end of file
+* `KOA_BILLING_CURRENCY_SYMBOL` (optional, default is '`$`'): sets a currency string to use to annotate costs on reports.
+* `KOA_INCLUDED_NAMESPACES` (optional, default is '`*`'): comma-separated list of namespaces to monitore. If any value is provided, only these namespaces will be accounted.
+* `KOA_EXCLUDED_NAMESPACES` (optional, default is ''): comma-separated list of namespaces to ignore. If any value is provided, these namespaces will be discarded. This options takes precedence over `KOA_INCLUDED_NAMESPACES`.

From 0d77fd62fed4053b34e74ddcb6b86c600738c686 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Thu, 9 Jun 2022 17:32:20 -0300
Subject: [PATCH 11/20] Dump resources.requests histogram analytics

---
 backend.py | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/backend.py b/backend.py
index 9f4e4e6..4629e3d 100644
--- a/backend.py
+++ b/backend.py
@@ -176,6 +176,15 @@ class RrdPeriod(enum.IntEnum):
                                                        ['namespace', 'resource'])
 }
 
+PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS = {
+    RrdPeriod.PERIOD_14_DAYS_SEC: prometheus_client.Gauge('koa_namespace_daily_requests',
+                                                          'Current daily resource reservation per namespace',
+                                                          ['namespace', 'resource']),
+    RrdPeriod.PERIOD_YEAR_SEC: prometheus_client.Gauge('koa_namespace_monthly_requests',
+                                                       'Current monthly resource reservation per namespace',
+                                                       ['namespace', 'resource'])
+}
+
 # create Flask application
 app = flask.Flask(__name__, static_url_path=KOA_CONFIG.static_content_location, template_folder='.')
 cors = CORS(app, resources={r"/dataset/*": {"origins": "127.0.0.1"}})
@@ -459,6 +468,7 @@ def extract_pods(self, data):
                 pod.nodeName = item['spec']['nodeName']
                 pod.cpuRequest = 0.0
                 pod.memRequest = 0.0
+                #TODO: extract initContainers
                 for _, container in enumerate(item.get('spec').get('containers')):
                     resources = container.get('resources', None)
                     if resources is not None:
@@ -696,9 +706,16 @@ def dump_histogram_analytics(dbfiles, period):
         usage_export = collections.defaultdict(list)
         usage_per_type_date = {}
         sum_usage_per_type_date = {}
+        requests_export = collections.defaultdict(list)
+        requests_per_type_date = {}
+        sum_requests_per_type_date = {}
         for _, db in enumerate(dbfiles):
             rrd = Rrd(db_files_location=KOA_CONFIG.db_location, dbname=db)
             current_periodic_usage = rrd.dump_histogram_data(period=period)
+
+            rrd = Rrd(db_files_location=KOA_CONFIG.db_location, dbname=KOA_CONFIG.usage_efficiency_db(db))
+            current_periodic_rf = rrd.dump_histogram_data(period=period)
+
             for res in [ResUsageType.CPU, ResUsageType.MEMORY]:
                 for date_key, usage_value in current_periodic_usage[res].items():
                     if usage_value > 0.0:
@@ -708,11 +725,26 @@ def dump_histogram_analytics(dbfiles, period):
                             usage_per_type_date[res][date_key] = {}
                         usage_per_type_date[res][date_key][db] = usage_value
 
+                        if res not in requests_per_type_date:
+                            requests_per_type_date[res] = collections.defaultdict(lambda: 0.0)
+                        if date_key not in requests_per_type_date[res]:
+                            requests_per_type_date[res][date_key] = {}
+
+                        rf = current_periodic_rf[res][date_key]
+                        if rf > 0.0:
+                            requests_per_type_date[res][date_key][db] = usage_value / rf
+                        else:
+                            requests_per_type_date[res][date_key][db] = 0.0
+
                         if db != KOA_CONFIG.db_billing_hourly_rate:
                             if res not in sum_usage_per_type_date:
                                 sum_usage_per_type_date[res] = collections.defaultdict(lambda: 0.0)
                             sum_usage_per_type_date[res][date_key] += usage_value
 
+                            if res not in sum_requests_per_type_date:
+                                sum_requests_per_type_date[res] = collections.defaultdict(lambda: 0.0)
+                            sum_requests_per_type_date[res][date_key] += usage_value
+
         for res, usage_data_bundle in usage_per_type_date.items():
             for date_key, db_usage_item in usage_data_bundle.items():
                 for db, usage_value in db_usage_item.items():
@@ -730,10 +762,28 @@ def dump_histogram_analytics(dbfiles, period):
                             PROMETHEUS_PERIODIC_USAGE_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
                                 usage_cost)
 
+                        requests_value = requests_per_type_date[res][date_key][db]
+                        requests_cost = round(requests_value, KOA_CONFIG.db_round_decimals)
+                        if KOA_CONFIG.cost_model == 'RATIO' or KOA_CONFIG.cost_model == 'CHARGE_BACK':
+                            requests_ratio = requests_value / sum_requests_per_type_date[res][date_key]
+                            requests_cost = round(100 * requests_ratio, KOA_CONFIG.db_round_decimals)
+                            if KOA_CONFIG.cost_model == 'CHARGE_BACK':
+                                requests_cost = round(
+                                    requests_ratio * usage_per_type_date[res][date_key][KOA_CONFIG.db_billing_hourly_rate],
+                                    KOA_CONFIG.db_round_decimals)
+                        requests_export[res].append('{"stack":"%s","requests":%f,"date":"%s"}' % (db, requests_cost, date_key))
+                        if Rrd.get_date_group(now_gmtime, period) == date_key:
+                            PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
+                                requests_cost)
+
         with open(str('%s/cpu_usage_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
             fd.write('[' + ','.join(usage_export[0]) + ']')
         with open(str('%s/memory_usage_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
             fd.write('[' + ','.join(usage_export[1]) + ']')
+        with open(str('%s/cpu_requests_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
+            fd.write('[' + ','.join(requests_export[0]) + ']')
+        with open(str('%s/memory_requests_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
+            fd.write('[' + ','.join(requests_export[1]) + ']')
 
 
 def pull_k8s(api_context):

From cf6b05712cc9d7174fbcd472142c5dee466ccf2e Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rodrigue.chakode@gmail.com>
Date: Sun, 12 Jun 2022 21:51:40 +0200
Subject: [PATCH 12/20] added visualization for requests

---
 index.html     |  8 +++++---
 js/frontend.js | 42 +++++++++++++++++++++++++++++-------------
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/index.html b/index.html
index c5b69e1..29a0663 100644
--- a/index.html
+++ b/index.html
@@ -144,9 +144,11 @@
         <div class="col-md-9">
             <legend>Usage Accounting <span class="accounting-cost-model"></span>
                 <select id="selected-cumulative-usage-type" onchange="FrontendApi.showCumulativeUsageByType()">
-                    <option value="default" disabled selected style="display:none;">Daily</option>
-                    <option value="daily">Daily</option>
-                    <option value="monthly">Monthly</option>
+                    <option value="default" disabled selected style="display:none;">Daily Usage</option>
+                    <option value="daily-usage">Daily Usage</option>
+                    <option value="daily-requests">Daily Requests</option>
+                    <option value="monthly-usage">Monthly Usage</option>
+                    <option value="monthly-requests">Monthly Requests</option>
                 </select>
             </legend>
         </div>
diff --git a/js/frontend.js b/js/frontend.js
index c908722..7af8904 100755
--- a/js/frontend.js
+++ b/js/frontend.js
@@ -503,16 +503,29 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
 
         function showCumulativeUsageByType() {
             cumulativeUsageType = $("#selected-cumulative-usage-type option:selected").val();
-            if (cumulativeUsageType === 'monthly') {
+            if (cumulativeUsageType === 'monthly-usage') {
                 $("#chart-block-daily").hide();
                 $("#chart-block-monthly").show();
-                refreshCumulativeChartByType(monthlyCpuUsageChart, 'CPU', 'monthly');
-                refreshCumulativeChartByType(monthlyMemoryUsageChart, 'Memory', 'monthly');
-            } else {
+                refreshCumulativeChartByType(monthlyCpuUsageChart, 'CPU', 'usage', 'monthly');
+                refreshCumulativeChartByType(monthlyMemoryUsageChart, 'Memory', 'usage', 'monthly');
+            } else if (cumulativeUsageType === 'monthly-requests') {
+                $("#chart-block-daily").hide();
+                $("#chart-block-monthly").show();
+                refreshCumulativeChartByType(monthlyCpuUsageChart, 'CPU', 'requests', 'monthly');
+                refreshCumulativeChartByType(monthlyMemoryUsageChart, 'Memory', 'requests', 'monthly');
+            } else if (cumulativeUsageType === 'daily-usage') {
                 $("#chart-block-monthly").hide();
                 $("#chart-block-daily").show();
-                refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'daily');
-                refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'daily');
+                refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'usage', 'daily');
+                refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'usage', 'daily');
+            } else if (cumulativeUsageType === 'daily-requests') {
+                $("#chart-block-monthly").hide();
+                $("#chart-block-daily").show();
+                refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'requests', 'daily');
+                refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'requests', 'daily');
+            } else {
+                $("#error-message").append('<li>unknown usage type (' + cumulativeUsageType + ')</li>');
+                $("#error-message-container").show();
             }
         }
 
@@ -660,21 +673,24 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
             });
         }
 
-        function refreshCumulativeChartByType(chart, resourceType, periodType) {
+        function refreshCumulativeChartByType(chart, resourceType, usageType, periodType) {
             let resourceTypeLowered = resourceType.toLowerCase();
             let DATASET_FILES = Object.freeze({
-                "daily": `${resourceTypeLowered}_usage_period_1209600.json`,
-                "monthly": `${resourceTypeLowered}_usage_period_31968000.json`,
+                "daily-usage": `${resourceTypeLowered}_usage_period_1209600.json`,
+                "daily-requests": `${resourceTypeLowered}_requests_period_1209600.json`,
+                "monthly-usage": `${resourceTypeLowered}_usage_period_31968000.json`,
+                "monthly-requests": `${resourceTypeLowered}_requests_period_31968000.json`,
             });
 
+            let dataSetKey = (periodType+'-'+usageType).toLowerCase();
             $.ajax({
                 type: "GET",
-                url: `${FrontendApi.DATA_DIR}/${DATASET_FILES[periodType]}`,
+                url: `${FrontendApi.DATA_DIR}/${DATASET_FILES[dataSetKey]}`,
                 dataType: 'json',
                 success: function (data) {
-                    installExporter(`consolidated-${resourceTypeLowered}-usage-png`, '', () => exportImage(chart, 'kopex-' + resourceTypeLowered + '-usage.png'));
-                    installExporter(`consolidated-${resourceTypeLowered}-usage-json`, 'kopex-' + resourceTypeLowered + '-usage.json', () => exportJSON(data));
-                    installExporter(`consolidated-${resourceTypeLowered}-usage-csv`, 'kopex-' + resourceTypeLowered + '-usage.csv', () => exportCSV(data));
+                    installExporter(`consolidated-${resourceTypeLowered}-png`, '', () => exportImage(chart, 'kopex-' + resourceTypeLowered + '.png'));
+                    installExporter(`consolidated-${resourceTypeLowered}-json`, 'kopex-' + resourceTypeLowered + '.json', () => exportJSON(data));
+                    installExporter(`consolidated-${resourceTypeLowered}-csv`, 'kopex-' + resourceTypeLowered  + '.csv', () => exportCSV(data));
 
                     updateStackedBarChart(
                         {"data": data},

From de0b0cd2e86d56a21dec062f4301109bc9473c14 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rodrigue.chakode@gmail.com>
Date: Mon, 13 Jun 2022 07:36:14 +0200
Subject: [PATCH 13/20] fixed lint

---
 backend.py | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)

diff --git a/backend.py b/backend.py
index 4629e3d..344f027 100644
--- a/backend.py
+++ b/backend.py
@@ -22,7 +22,7 @@
 import threading
 import time
 import traceback
-import urllib, urllib3
+import urllib
 from typing import Any, List
 
 import flask
@@ -35,12 +35,15 @@
 
 import rrdtool
 
+import urllib3
+
 from waitress import serve as waitress_serve
 
 import werkzeug.middleware.dispatcher as wsgi
 
 urllib3.disable_warnings()
 
+
 def create_directory_if_not_exists(path):
     try:
         os.makedirs(path)
@@ -70,8 +73,8 @@ class Config:
     k8s_ssl_cacert = os.getenv('KOA_K8S_CACERT', None)
     k8s_ssl_client_cert = os.getenv('KOA_K8S_AUTH_CLIENT_CERT', 'NO_ENV_CLIENT_CERT')
     k8s_ssl_client_cert_key = os.getenv('KOA_K8S_AUTH_CLIENT_CERT_KEY', 'NO_ENV_CLIENT_CERT_CERT')
-    included_namespaces = [ i for i in os.getenv('KOA_INCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
-    excluded_namespaces = [ i for i in os.getenv('KOA_EXCLUDED_NAMESPACES', '').replace(' ','').split(',') if i ]
+    included_namespaces = [i for i in os.getenv('KOA_INCLUDED_NAMESPACES', '').replace(' ', ',').split(',') if i]
+    excluded_namespaces = [i for i in os.getenv('KOA_EXCLUDED_NAMESPACES', '').replace(' ', ',').split(',') if i]
 
     def __init__(self):
         self.load_rbac_auth_token()
@@ -103,16 +106,18 @@ def __init__(self):
 
     @staticmethod
     def match(items, pattern):
-        return any([ fnmatch.fnmatch(i, pattern) for i in items ])
+        return any([fnmatch.fnmatch(i, pattern) for i in items])
 
     @staticmethod
     def allow_namespace(namespace):
         if KOA_CONFIG.match(KOA_CONFIG.excluded_namespaces, namespace):
             return False
 
-        return len(KOA_CONFIG.included_namespaces) == 0 or \
-                '*' in KOA_CONFIG.included_namespaces or \
-                KOA_CONFIG.match(KOA_CONFIG.included_namespaces, namespace)
+        no_namespace_included = len(KOA_CONFIG.included_namespaces) == 0
+        all_namespaces_enabled = '*' in KOA_CONFIG.included_namespaces
+        namespace_matched = KOA_CONFIG.match(KOA_CONFIG.included_namespaces, namespace)
+
+        return no_namespace_included or all_namespaces_enabled or namespace_matched
 
     def load_rbac_auth_token(self):
         try:
@@ -468,7 +473,7 @@ def extract_pods(self, data):
                 pod.nodeName = item['spec']['nodeName']
                 pod.cpuRequest = 0.0
                 pod.memRequest = 0.0
-                #TODO: extract initContainers
+                # TODO: extract initContainers
                 for _, container in enumerate(item.get('spec').get('containers')):
                     resources = container.get('resources', None)
                     if resources is not None:
@@ -763,18 +768,19 @@ def dump_histogram_analytics(dbfiles, period):
                                 usage_cost)
 
                         requests_value = requests_per_type_date[res][date_key][db]
-                        requests_cost = round(requests_value, KOA_CONFIG.db_round_decimals)
+                        req_cost = round(requests_value, KOA_CONFIG.db_round_decimals)
                         if KOA_CONFIG.cost_model == 'RATIO' or KOA_CONFIG.cost_model == 'CHARGE_BACK':
-                            requests_ratio = requests_value / sum_requests_per_type_date[res][date_key]
-                            requests_cost = round(100 * requests_ratio, KOA_CONFIG.db_round_decimals)
+                            req_ratio = requests_value / sum_requests_per_type_date[res][date_key]
+                            req_cost = round(100 * req_ratio, KOA_CONFIG.db_round_decimals)
                             if KOA_CONFIG.cost_model == 'CHARGE_BACK':
-                                requests_cost = round(
-                                    requests_ratio * usage_per_type_date[res][date_key][KOA_CONFIG.db_billing_hourly_rate],
+                                req_cost = round(
+                                    req_ratio * usage_per_type_date[res][date_key][KOA_CONFIG.db_billing_hourly_rate],
                                     KOA_CONFIG.db_round_decimals)
-                        requests_export[res].append('{"stack":"%s","requests":%f,"date":"%s"}' % (db, requests_cost, date_key))
+                        requests_export[res].append('{"stack":"%s","requests":%f,"date":"%s"}'
+                                                    % (db, req_cost, date_key))
                         if Rrd.get_date_group(now_gmtime, period) == date_key:
                             PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
-                                requests_cost)
+                                req_cost)
 
         with open(str('%s/cpu_usage_period_%d.json' % (KOA_CONFIG.frontend_data_location, period)), 'w') as fd:
             fd.write('[' + ','.join(usage_export[0]) + ']')

From c3a54d112d2d8f8a270cc1bd015e7cbff8f99bc2 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rodrigue.chakode@gmail.com>
Date: Mon, 13 Jun 2022 14:01:40 +0200
Subject: [PATCH 14/20] chore: use uniform data structure for usage and
 requests

---
 backend.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/backend.py b/backend.py
index 344f027..bbcb9b1 100644
--- a/backend.py
+++ b/backend.py
@@ -767,16 +767,16 @@ def dump_histogram_analytics(dbfiles, period):
                             PROMETHEUS_PERIODIC_USAGE_EXPORTERS[period].labels(db, ResUsageType(res).name).set(
                                 usage_cost)
 
-                        requests_value = requests_per_type_date[res][date_key][db]
-                        req_cost = round(requests_value, KOA_CONFIG.db_round_decimals)
+                        req_value = requests_per_type_date[res][date_key][db]
+                        req_cost = round(req_value, KOA_CONFIG.db_round_decimals)
                         if KOA_CONFIG.cost_model == 'RATIO' or KOA_CONFIG.cost_model == 'CHARGE_BACK':
-                            req_ratio = requests_value / sum_requests_per_type_date[res][date_key]
+                            req_ratio = req_value / sum_requests_per_type_date[res][date_key]
                             req_cost = round(100 * req_ratio, KOA_CONFIG.db_round_decimals)
                             if KOA_CONFIG.cost_model == 'CHARGE_BACK':
                                 req_cost = round(
                                     req_ratio * usage_per_type_date[res][date_key][KOA_CONFIG.db_billing_hourly_rate],
                                     KOA_CONFIG.db_round_decimals)
-                        requests_export[res].append('{"stack":"%s","requests":%f,"date":"%s"}'
+                        requests_export[res].append('{"stack":"%s","usage":%f,"date":"%s"}'
                                                     % (db, req_cost, date_key))
                         if Rrd.get_date_group(now_gmtime, period) == date_key:
                             PROMETHEUS_PERIODIC_REQUESTS_EXPORTERS[period].labels(db, ResUsageType(res).name).set(

From 9171d4a7a6140ad797d336d7fc7d51e1126bea33 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rodrigue.chakode@gmail.com>
Date: Mon, 13 Jun 2022 14:02:35 +0200
Subject: [PATCH 15/20] fix: set daily usage as default display for accounting

---
 js/frontend.js | 16 +++++++---------
 1 file changed, 7 insertions(+), 9 deletions(-)

diff --git a/js/frontend.js b/js/frontend.js
index 7af8904..1d30a81 100755
--- a/js/frontend.js
+++ b/js/frontend.js
@@ -503,6 +503,7 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
 
         function showCumulativeUsageByType() {
             cumulativeUsageType = $("#selected-cumulative-usage-type option:selected").val();
+            console.log(cumulativeUsageType);
             if (cumulativeUsageType === 'monthly-usage') {
                 $("#chart-block-daily").hide();
                 $("#chart-block-monthly").show();
@@ -513,19 +514,16 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
                 $("#chart-block-monthly").show();
                 refreshCumulativeChartByType(monthlyCpuUsageChart, 'CPU', 'requests', 'monthly');
                 refreshCumulativeChartByType(monthlyMemoryUsageChart, 'Memory', 'requests', 'monthly');
-            } else if (cumulativeUsageType === 'daily-usage') {
-                $("#chart-block-monthly").hide();
-                $("#chart-block-daily").show();
-                refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'usage', 'daily');
-                refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'usage', 'daily');
             } else if (cumulativeUsageType === 'daily-requests') {
                 $("#chart-block-monthly").hide();
                 $("#chart-block-daily").show();
                 refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'requests', 'daily');
                 refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'requests', 'daily');
-            } else {
-                $("#error-message").append('<li>unknown usage type (' + cumulativeUsageType + ')</li>');
-                $("#error-message-container").show();
+            } else { // handle as daily-usage
+                $("#chart-block-monthly").hide();
+                $("#chart-block-daily").show();
+                refreshCumulativeChartByType(dailyCpuUsageChart, 'CPU', 'usage', 'daily');
+                refreshCumulativeChartByType(dailyMemoryUsageChart, 'Memory', 'usage', 'daily');
             }
         }
 
@@ -697,7 +695,7 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
                         chart,
                         `js-chart-${periodType}-${resourceTypeLowered}-usage`,
                         `${resourceType} consumption`,
-                        `${periodType} ${resourceType} Usage`);
+                        `${periodType} ${resourceType} ${usageType}`);
                 },
                 error: function (xhr, ajaxOptions, thrownError) {
                     $("#error-message").append(`<li>error ${xhr.status} downloading data file ${DATASET_FILES[periodType]}</li>`);

From 2ddaa64d01c491c91b8bd1668bb1e1120daee017 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rchakode@users.noreply.github.com>
Date: Mon, 13 Jun 2022 15:01:41 +0200
Subject: [PATCH 16/20] Revert "ci: trigger build and lint on commit and pr"

This reverts commit c75f6a1ceff404456bfa57d1ff639f47a3d89028.
---
 .github/workflows/pythonpackage.yml | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/.github/workflows/pythonpackage.yml b/.github/workflows/pythonpackage.yml
index fb895af..fb38a5e 100644
--- a/.github/workflows/pythonpackage.yml
+++ b/.github/workflows/pythonpackage.yml
@@ -1,12 +1,6 @@
 name: Build
 
-on:
-  push:
-    branches:
-    - '**'
-  pull_request:
-    branches:
-    - main
+on: [push]
 
 jobs:
   build:

From 28a8165b02ce946e00efaf37faf64d0df2543968 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Mon, 13 Jun 2022 10:49:43 -0300
Subject: [PATCH 17/20] remove dup urllib3 import

---
 backend.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/backend.py b/backend.py
index eba57e7..bbcb9b1 100644
--- a/backend.py
+++ b/backend.py
@@ -22,7 +22,7 @@
 import threading
 import time
 import traceback
-import urllib, urllib3
+import urllib
 from typing import Any, List
 
 import flask

From 2e6ab0e47140fe29ba8095fa1752989ffa634626 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Mon, 13 Jun 2022 10:50:05 -0300
Subject: [PATCH 18/20] installs pytest into tox testenv

---
 tox.ini | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tox.ini b/tox.ini
index 6148b3a..225a0db 100644
--- a/tox.ini
+++ b/tox.ini
@@ -4,6 +4,7 @@ skipsdist = true
 
 [testenv]
 deps = -rrequirements.txt
+       pytest
 commands = pytest
 
 [testenv:lint]
@@ -15,6 +16,7 @@ deps = flake8
        flake8-rst-docstrings
        flake8-logging-format
        pydocstyle < 4.0.0
+       pytest
 
 commands = flake8 {posargs}
 

From 9bc8c0408adb038ebe6b39e2a032231346720a38 Mon Sep 17 00:00:00 2001
From: Mateus Caruccio <mateus.caruccio@getupcloud.com>
Date: Mon, 13 Jun 2022 17:03:12 -0300
Subject: [PATCH 19/20] Fix consolidated exporting

- Fix export link selector;
- Save filenames based on resource type and dataset key.
---
 js/frontend.js | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/js/frontend.js b/js/frontend.js
index 1d30a81..6011f69 100755
--- a/js/frontend.js
+++ b/js/frontend.js
@@ -681,14 +681,15 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
             });
 
             let dataSetKey = (periodType+'-'+usageType).toLowerCase();
+            let filenamePrefix = 'kopex-' + resourceTypeLowered + '-' + dataSetKey;
             $.ajax({
                 type: "GET",
                 url: `${FrontendApi.DATA_DIR}/${DATASET_FILES[dataSetKey]}`,
                 dataType: 'json',
                 success: function (data) {
-                    installExporter(`consolidated-${resourceTypeLowered}-png`, '', () => exportImage(chart, 'kopex-' + resourceTypeLowered + '.png'));
-                    installExporter(`consolidated-${resourceTypeLowered}-json`, 'kopex-' + resourceTypeLowered + '.json', () => exportJSON(data));
-                    installExporter(`consolidated-${resourceTypeLowered}-csv`, 'kopex-' + resourceTypeLowered  + '.csv', () => exportCSV(data));
+                    installExporter(`consolidated-${resourceTypeLowered}-usage-png`, '', () => exportImage(chart, filenamePrefix + '.png'));
+                    installExporter(`consolidated-${resourceTypeLowered}-usage-json`, filenamePrefix + '.json', () => exportJSON(data));
+                    installExporter(`consolidated-${resourceTypeLowered}-usage-csv`, filenamePrefix + '.csv', () => exportCSV(data));
 
                     updateStackedBarChart(
                         {"data": data},
@@ -773,4 +774,4 @@ define(['jquery', 'bootstrap', 'bootswatch', 'd3Selection', 'stackedAreaChart',
         FrontendApi.showSelectedUsageTrendType = showUsageTrendByType;
         FrontendApi.showCumulativeUsageByType = showCumulativeUsageByType;
     }
-);
\ No newline at end of file
+);

From 2d88c86dea339c370744262e92875b148f93bb76 Mon Sep 17 00:00:00 2001
From: Rodrigue Chakode <rodrigue.chakode@gmail.com>
Date: Sun, 26 Jun 2022 23:53:34 +0200
Subject: [PATCH 20/20] added docs related to requested resources accouting

---
 README.md                              | 29 +++++++++++++-------------
 docs/built-in-dashboards-and-charts.md | 15 +++++++------
 2 files changed, 24 insertions(+), 20 deletions(-)

diff --git a/README.md b/README.md
index 24410c9..068f3c4 100644
--- a/README.md
+++ b/README.md
@@ -12,18 +12,18 @@
 - [Design Fundamentals](#design-fundamentals)
 - [License](#license)
 - [Support & Contributions](#support--contributions)
- 
+
 # Overview
 `kube-opex-analytics` (literally *Kubernetes Opex Analytics*) is a Kubernetes usage accounting and analytics tool to help organizations track the resources being consumed by their Kubernetes clusters over time (hourly, daily, monthly). The purpose of `kube-opex-analytics` is to help prevent overpaying. Indeed, it provides insightful usage analytics metrics and charts, that engineering and financial teams can use as key indicators to take appropriate cost optimization decisions.
 
 Key features:
 
- * **Usage accounting and trends per namespace.** This allows assessing what capacities each namespace is consuming over various period of time (hourly, daily, monthly).
- * **Accounting of non-allocatable capacities.** At node and cluster levels, `kube-opex-analytics` tracks and consolidates the share of non-allocatable capacities and highlights them against usable capacities (i.e. capacities used by actual application workloads). In contrary to usable capacities, non-allocatable capacities are dedicated to Kubernetes operations (OS, kubelets, etc).
- * **Cluster usage accounting and capacity planning.** This feature makes it easy to account and visualize capacities consumed on a cluster, globally, instantly and over time.
- * **Usage/requests efficiency.** Based on hourly-consolidated trends, this functionality helps know how efficient resource requests set on Kubernetes workloads are, compared against the actual resource usage over time.
- * **Cost allocation and charge back analytics:** automatic processing and visualization of resource usage accounting per namespace on daily and monthly periods.
- * **Insightful and extensible visualization.** `kube-opex-analytics` enables built-in analytics dashboards, as well as a native Prometheus exporter that exposes its analytics metrics for third-party visualization tools like Grafana.
+* **Consumption hourly trends, daily and monthly accounting per namespace.** This feature provides analytics metrics _tracking both actual usage and requested capacities_ over time. Metrics are namespaced-based, collected every 5 minutes, consolidated on a hourly basis for trends, from which daily and monthly accounting is processed.
+* **Accounting of non-allocatable capacities.** At node and cluster levels, `kube-opex-analytics` tracks and consolidates the share of non-allocatable capacities and highlights them against usable capacities (i.e. capacities used by actual application workloads). In contrary to usable capacities, non-allocatable capacities are dedicated to Kubernetes operations (OS, kubelets, etc).
+* **Cluster usage accounting and capacity planning.** This feature makes it easy to account and visualize capacities consumed on a cluster, globally, instantly and over time.
+* **Usage/requests efficiency.** Based on hourly-consolidated trends, this functionality helps know how efficient resource requests set on Kubernetes workloads are, compared against the actual resource usage over time.
+* **Cost allocation and charge back analytics:** automatic processing and visualization of resource usage accounting per namespace on daily and monthly periods.
+* **Insightful and extensible visualization.** `kube-opex-analytics` enables built-in analytics dashboards, as well as a native Prometheus exporter that exposes its analytics metrics for third-party visualization tools like Grafana.
 
 
 ![kube-opex-analytics-overview](screenshots/kube-opex-analytics-demo.gif)
@@ -31,16 +31,16 @@ Key features:
 Read the [design fundamentals](./docs/design-fundamentals.md) documentation to learn more concepts and implementation decisions.
 
 # Getting Started
-  * [Installation on Kubernetes](./docs/installation-on-kubernetes.md)
-  * [Installation on Docker](./docs/installation-on-docker.md)
-  * [Built-in dashboards and charts](./docs/built-in-dashboards-and-charts.md)
-  * [Prometheus Exporter and Grafana Dashboards](./docs/prometheus-exporter-grafana-dashboard.md)
-  * [Configuration Settings](./docs/configuration-settings.md)
-  * [Centralized multi Kubernetes clusters analytics](./docs/multi-cluster-analytics.md)
+* [Installation on Kubernetes](./docs/installation-on-kubernetes.md)
+* [Installation on Docker](./docs/installation-on-docker.md)
+* [Built-in dashboards and charts](./docs/built-in-dashboards-and-charts.md)
+* [Prometheus Exporter and Grafana Dashboards](./docs/prometheus-exporter-grafana-dashboard.md)
+* [Configuration Settings](./docs/configuration-settings.md)
+* [Centralized multi Kubernetes clusters analytics](./docs/multi-cluster-analytics.md)
 
 # Design Fundamentals
 Checkout the [Design Fundamentals](./docs/design-fundamentals.md) documentation to learn more about `kube-opex-analytics`, it introduces concepts and implementation decisions.
- 
+
 > **Multi-cluster analytics:** `kube-opex-analytics` tracks the usage for a single instance of Kubernetes. For a centralized multi-Kubernetes usage analytics, you may have to consider our [Krossboard](https://krossboard.app/) product. Watch a [demo video](https://youtu.be/lfkUIREDYDY).
 
 # License
@@ -54,3 +54,4 @@ Use this [link to submit issues or improvement ideas](https://github.com/rchakod
 To contribute bug patches or new features, please submit a [Pull Request](https://github.com/rchakode/kube-opex-analytics/pulls).
 
 Contributions are accepted subject that the code and documentation be released under the terms of Apache 2.0 License.
+
diff --git a/docs/built-in-dashboards-and-charts.md b/docs/built-in-dashboards-and-charts.md
index 86e84db..67e4342 100644
--- a/docs/built-in-dashboards-and-charts.md
+++ b/docs/built-in-dashboards-and-charts.md
@@ -4,7 +4,7 @@ This section describes the built-in dashboards and charts provided by `kube-opex
 - [Built-in Dashboards and Charts of kube-opex-analytics](#built-in-dashboards-and-charts-of-kube-opex-analytics)
   - [Hourly Consolidated Usage Trends (7 days)](#hourly-consolidated-usage-trends-7-days)
   - [Hourly Usage/Requests Efficiency (7 days)](#hourly-usagerequests-efficiency-7-days)
-  - [Daily CPU and Memory Usage (14 days)](#daily-cpu-and-memory-usage-14-days)
+  - [Daily Consumption Accounting (14 days)](#daily-consumption-accounting-14-days)
   - [Monthly CPU and Memory Usage (12 months)](#monthly-cpu-and-memory-usage-12-months)
   - [Nodes' Occupation by Pods](#nodes-occupation-by-pods)
   - [Export Charts and Datasets (PNG, CSV, JSON)](#export-charts-and-datasets-png-csv-json)
@@ -30,14 +30,17 @@ The date filter can be used to zoom out/in on a specific time range.
 
 These charts are based on data consolidated hourly thanks to sample metrics collected every five minutes from Kubernetes. 
 
-## Daily CPU and Memory Usage (14 days)
-For the different namespaces discovered in the Kubernetes cluster, these charts show daily cumulative usage for CPU and memory resources during the last 2 weeks. 
+## Daily Consumption Accounting (14 days)
+The daily accounting charts are provided per namespace for CPU and Memory resources and cover the last 14 days (2 weeks).
 
-![](../screenshots/sample-two-weeks-daily-usage.png)
+According to the [selected accounting model (](design-fundamentals.md#usage-accounting-models), the charts display one of the following metrics :
 
-These charts are based on data consolidated hourly thanks to sample metrics collected every five minutes from Kubernetes. 
+* Daily cumulative sum of actual hourly consumption per namespace.
+* Daily cumulative sum of the maximum between the actual hourly consumption and the requested capacities.
+* Daily cumulative sum of actual hourly computedd from an actual cluster cost set statically based on a fixed hourly rate, or determinated dynamically from allocated resources on public clouds (nodes, storage, etc.).
+  
+![](../screenshots/sample-two-weeks-daily-usage.png)
 
-Depending on the [selected accounting model](design-fundamentals.md#usage-accounting-models), the values on these charts can be actual costs (`CHARGE_BACK` model), cumulative usage (sum of hourly consolidated usage, `CUMULATIVE_RATIO` model), or a percentage of the global cluster usage (`CHARGE_BACK` model, `100%` means the global cluster usage).
 
 ## Monthly CPU and Memory Usage (12 months)
 For the different namespaces discovered in the Kubernetes cluster, these charts show monthly cumulative usage for CPU and memory resources during the last 12 months.