8000 Feature/089 namespace on env vars with backward compatibility (#191) · python-microservices/pyms@995e476 · GitHub
[go: up one dir, main page]
Skip to content

Commit 995e476

Browse files
rapejimrapejim
authored
Feature/089 namespace on env vars with backward compatibility (#191)
* feature: Added namespace to environment variables and legacy versions * feature: Added warnings at use case of old environment variables names * feature: Added backward compatibility at use case of old environment variables names * docs: Added namespace to environment variables on comments and docs * style: changed warning messages for env. vars. deprecated * style: Added new util for apply color or style to text * fix: logic for backward compatibility
1 parent 03a033a commit 995e476

18 files changed

+227
-29
lines changed

‎docs/configuration.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
## Environments variables of PyMS:
44

5-
**CONFIGMAP_FILE**: The path to the configuration file. By default, PyMS searches for the configuration file in your
5+
**PYMS_CONFIGMAP_FILE**: The path to the configuration file. By default, PyMS searches for the configuration file in your
66
current folder with the name "config.yml"
7-
**KEY_FILE**: The path to the key file to decrypt your configuration. By default, PyMS searches for the configuration file in your
7+
**PYMS_KEY_FILE**: The path to the key file to decrypt your configuration. By default, PyMS searches for the configuration file in your
88
current folder with the name "key.key"
99

1010
## Create configuration
@@ -136,7 +136,7 @@ API = Api(
136136

137137
## Looking for Configuration file with Kubernetes Configmaps
138138
By default, the Microservice class searches for a config.yml in the same path. You can set a different route or set a json file.
139-
To change this path, you must define an environment variable called `CONFIGMAP_FILE`.
139+
To change this path, you must define an environment variable called `PYMS_CONFIGMAP_FILE`.
140140

141141
This way of looking for the configuration is useful when you work with Docker and Kubernetes. For example, you could integrate
142142
a configmap of Kubernetes, with this microservice and a deployment with:
@@ -154,7 +154,7 @@ spec:
154154
- name: my-microservice
155155
image: ...
156156
env:
157-
- name: CONFIGMAP_FILE
157+
- name: PYMS_CONFIGMAP_FILE
158158
value: "/usr/share/microservice/config.yaml"
159159
160160
volumeMounts:

‎docs/encrypt_decryt_configuration.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ Move your key, for example, to `mv key.key /home/my_user/keys/myproject.key`
5757
then, store the key in a environment variable with:
5858

5959
```bash
60-
export KEY_FILE=/home/my_user/keys/myproject.key
60+
export PYMS_KEY_FILE=/home/my_user/keys/myproject.key
6161
```
6262

6363
## 3. Encrypt your information and store it in config
@@ -89,7 +89,7 @@ can find the answer
8989
## 4. Decrypt from your config file
9090

9191
Pyms knows if a variable is encrypted if this var start with the prefix `enc_` or `ENC_`. PyMS searches for your key file
92-
in the `KEY_FILE` env variable and decrypts this value to store it in the same variable without the `enc_` prefix,
92+
in the `PYMS_KEY_FILE` env variable and decrypts this value to store it in the same variable without the `enc_` prefix,
9393
for example,
9494

9595
```yaml

‎docs/ms_class.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ pyms:
3333
Check the [Configuration](configuration.md) section to learn how to create a configuration file.
3434
3535
The `Microservice` class searches for a `config.yml` in the directory you pass in `path` parameter or looks for the file in
36-
`CONFIGMAP_FILE` env var.
36+
`PYMS_CONFIGMAP_FILE` env var.
3737

3838
Each keyword in our configuration block, can be accessed in our Microservice object through the attribute `config`.
3939

‎examples/mininum_microservice_docker/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ RUN apk add --update curl gcc g++ git libffi-dev openssl-dev python3-dev build-b
44
&& rm -rf /var/cache/apk/*
55

66
ENV PYTHONUNBUFFERED=1 APP_HOME=/microservice/
7-
ENV CONFIGMAP_FILE="$APP_HOME"config-docker.yml
7+
ENV PYMS_CONFIGMAP_FILE="$APP_HOME"config-docker.yml
88

99
RUN mkdir $APP_HOME && adduser -S -D -H python
1010
RUN chown -R python $APP_HOME

‎pylintrc

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -291,8 +291,7 @@ min-similarity-lines=4
291291

292292
# List of note tags to take in consideration, separated by a comma.
293293
notes=FIXME,
294-
XXX,
295-
TODO
294+
XXX
296295

297296

298297
[TYPECHECK]

‎pyms/cmd/main.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
from pyms.crypt.fernet import Crypt
1010
from pyms.flask.services.swagger import merge_swagger_file
11-
from pyms.utils import check_package_exists, import_from
11+
from pyms.utils import check_package_exists, import_from, utils
1212

1313

1414
class Command:
@@ -117,15 +117,15 @@ def run(self):
117117

118118
@staticmethod
119119
def print_ok(msg=""):
120-
print('\033[92m\033[1m ' + msg + ' \033[0m\033[0m')
120+
print(utils.colored_text(msg, utils.Colors.BRIGHT_GREEN, True))
121121

122122
def print_verbose(self, msg=""): # pragma: no cover
123123
if self.verbose:
124124
print(msg)
125125

126126
@staticmethod
127127
def print_error(msg=""): # pragma: no cover
128-
print('\033[91m\033[1m ' + msg + ' \033[0m\033[0m')
128+
print(utils.colored_text(msg, utils.Colors.BRIGHT_RED, True))
129129

130130
def exit_with_error(self, msg=""): # pragma: no cover
131131
self.print_error(msg)

‎pyms/config/conf.py

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
1+
import logging
2+
import os
3+
14
from pyms.config.confile import ConfFile
2-
from pyms.constants import PYMS_CONFIG_WHITELIST_KEYWORDS
5+
from pyms.constants import PYMS_CONFIG_WHITELIST_KEYWORDS, CONFIGMAP_FILE_ENVIRONMENT_LEGACY, \
6+
CONFIGMAP_FILE_ENVIRONMENT, CRYPT_FILE_KEY_ENVIRONMENT, CRYPT_FILE_KEY_ENVIRONMENT_LEGACY, LOGGER_NAME
37
from pyms.exceptions import ServiceDoesNotExistException, ConfigErrorException, AttrDoesNotExistException
8+
from pyms.utils import utils
9+
10+
logger = logging.getLogger(LOGGER_NAME)
411

512

613
def get_conf(*args, **kwargs):
@@ -42,6 +49,7 @@ def get_conf(*args, **kwargs):
4249

4350

4451
def validate_conf(*args, **kwargs):
52+
4553
config = ConfFile(*args, **kwargs)
4654
is_config_ok = True
4755
try:
@@ -104,3 +112,35 @@ def validate_conf(*args, **kwargs):
104112
config:
105113
DEBUG: true
106114
TESTING: true""".format(wrong_keywords))
115+
116+
# TODO Remove temporally deprecated warnings on future versions
117+
__verify_deprecated_env_variables(config)
118+
119+
120+
def __verify_deprecated_env_variables(config):
121+
env_var_duplicated = "IMPORTANT: If you are using \"{}\" environment variable, \"{}\" value will be ignored."
122+
env_var_deprecated = "IMPORTANT: \"{}\" environment variable is deprecated on this version, use \"{}\" instead."
123+
124+
if os.getenv(CONFIGMAP_FILE_ENVIRONMENT_LEGACY) is not None:
125+
if os.getenv(CONFIGMAP_FILE_ENVIRONMENT) is not None:
126+
msg = env_var_duplicated.format(CONFIGMAP_FILE_ENVIRONMENT, CONFIGMAP_FILE_ENVIRONMENT_LEGACY)
127+
else:
128+
msg = env_var_deprecated.format(CONFIGMAP_FILE_ENVIRONMENT_LEGACY, CONFIGMAP_FILE_ENVIRONMENT)
129+
try:
130+
if config.pyms.config.DEBUG:
131+
msg = utils.colored_text(msg, utils.Colors.BRIGHT_YELLOW, True)
132+
except AttrDoesNotExistException:
133+
pass
134+
logger.warning(msg)
135+
136+
if os.getenv(CRYPT_FILE_KEY_ENVIRONMENT_LEGACY) is not None:
137+
if os.getenv(CRYPT_FILE_KEY_ENVIRONMENT) is not None:
138+
msg = env_var_duplicated.format(CRYPT_FILE_KEY_ENVIRONMENT, CRYPT_FILE_KEY_ENVIRONMENT_LEGACY)
139+
else:
140+
msg = env_var_deprecated.format(CRYPT_FILE_KEY_ENVIRONMENT_LEGACY, CRYPT_FILE_KEY_ENVIRONMENT)
141+
try:
142+
if config.pyms.config.DEBUG:
143+
msg = utils.colored_text(msg, utils.Colors.BRIGHT_YELLOW, True)
144+
except AttrDoesNotExistException:
145+
pass
146+
logger.warning(msg)

‎pyms/config/confile.py

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
"""Module to read yaml or json conf"""
22
import logging
3+
import os
34
import re
45
from typing import Dict, Union, Text, Tuple, Iterable
56

67
import anyconfig
78

8-
from pyms.constants import CONFIGMAP_FILE_ENVIRONMENT, LOGGER_NAME, DEFAULT_CONFIGMAP_FILENAME
9+
from pyms.constants import CONFIGMAP_FILE_ENVIRONMENT, LOGGER_NAME, DEFAULT_CONFIGMAP_FILENAME, \
10+
CONFIGMAP_FILE_ENVIRONMENT_LEGACY
911
from pyms.exceptions import AttrDoesNotExistException, ConfigDoesNotFoundException
1012
from pyms.utils.files import LoadFile
1113

@@ -14,7 +16,7 @@
1416

1517
class ConfFile(dict):
1618
"""Recursive get configuration from dictionary, a config file in JSON or YAML format from a path or
17-
`CONFIGMAP_FILE` environment variable.
19+
`PYMS_CONFIGMAP_FILE` environment variable.
1820
**Atributes:**
1921
* path: Path to find the `DEFAULT_CONFIGMAP_FILENAME` and `DEFAULT_KEY_FILENAME` if use encrypted vars
2022
* empty_init: Allow blank variables
@@ -26,15 +28,18 @@ class ConfFile(dict):
2628
def __init__(self, *args, **kwargs):
2729
"""
2830
Get configuration from a dictionary(variable `config`), from path (variable `path`) or from
29-
environment with the constant `CONFIGMAP_FILE`
31+
environment with the constant `PYMS_CONFIGMAP_FILE`
3032
Set the configuration as upper case to inject the keys in flask config. Flask search for uppercase keys in
3133
`app.config.from_object`
3234
```python
3335
if key.isupper():
3436
self[key] = getattr(obj, key)
3537
```
3638
"""
37-
self._loader = LoadFile(kwargs.get("path"), CONFIGMAP_FILE_ENVIRONMENT, DEFAULT_CONFIGMAP_FILENAME)
39+
# TODO Remove temporally backward compatibility on future versions
40+
configmap_file_env = self.__get_updated_configmap_file_env() # Temporally backward compatibility
41+
42+
self._loader = LoadFile(kwargs.get("path"), configmap_file_env, DEFAULT_CONFIGMAP_FILENAME)
3843
self._crypt_cls = kwargs.get("crypt")
3944
if self._crypt_cls:
4045
self._crypt = self._crypt_cls(path=kwargs.get("path"))
@@ -125,3 +130,10 @@ def reload(self):
125130

126131
def __setattr__(self, name, value, *args, **kwargs):
127132
super().__setattr__(name, value)
133+
134+
@staticmethod
135+
def __get_updated_configmap_file_env() -> str:
136+
result = CONFIGMAP_FILE_ENVIRONMENT
137+
if (os.getenv(CONFIGMAP_FILE_ENVIRONMENT_LEGACY) is not None) and (os.getenv(CONFIGMAP_FILE_ENVIRONMENT) is None):
138+
result = CONFIGMAP_FILE_ENVIRONMENT_LEGACY
139+
return result

‎pyms/constants.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1-
CONFIGMAP_FILE_ENVIRONMENT = "CONFIGMAP_FILE"
1+
CONFIGMAP_FILE_ENVIRONMENT = "PYMS_CONFIGMAP_FILE"
2+
CONFIGMAP_FILE_ENVIRONMENT_LEGACY = "CONFIGMAP_FILE"
23

34
DEFAULT_CONFIGMAP_FILENAME = "config.yml"
45

5-
CRYPT_FILE_KEY_ENVIRONMENT = "KEY_FILE"
6+
CRYPT_FILE_KEY_ENVIRONMENT = "PYMS_KEY_FILE"
7+
CRYPT_FILE_KEY_ENVIRONMENT_LEGACY = "KEY_FILE"
68

79
DEFAULT_KEY_FILENAME = "key.key"
810

‎pyms/crypt/fernet.py

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,17 @@
77
from cryptography.hazmat.primitives import hashes
88
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
99

10-
from pyms.constants import CRYPT_FILE_KEY_ENVIRONMENT, DEFAULT_KEY_FILENAME
10+
from pyms.constants import CRYPT_FILE_KEY_ENVIRONMENT, DEFAULT_KEY_FILENAME, CRYPT_FILE_KEY_ENVIRONMENT_LEGACY
1111
from pyms.crypt.driver import CryptAbstract
1212
from pyms.exceptions import FileDoesNotExistException
1313
from pyms.utils.files import LoadFile
1414

1515

1616
class Crypt(CryptAbstract):
1717
def __init__(self, *args, **kwargs):
18-
self._loader = LoadFile(kwargs.get("path"), CRYPT_FILE_KEY_ENVIRONMENT, DEFAULT_KEY_FILENAME)
18+
# TODO Remove temporally backward compatibility on future versions
19+
crypt_file_key_env = self.__get_updated_crypt_file_key_env() # Temporally backward compatibility
20+
self._loader = LoadFile(kwargs.get("path"), crypt_file_key_env, DEFAULT_KEY_FILENAME)
1921
super().__init__(*args, **kwargs)
2022

2123
def generate_key(self, password: Text, write_to_file: bool = False) -> bytes:
@@ -36,9 +38,11 @@ def generate_key(self, password: Text, write_to_file: bool = False) -> bytes:
3638
def read_key(self):
3739
key = self._loader.get_file()
3840
if not key:
41+
# TODO Remove temporally backward compatibility on future versions
42+
crypt_file_key_env = self.__get_updated_crypt_file_key_env() # Temporally backward compatibility
3943
raise FileDoesNotExistException(
4044
"Decrypt key {} not exists. You must set a correct env var {} "
41-
"or run `pyms crypt create-key` command".format(self._loader.path, CRYPT_FILE_KEY_ENVIRONMENT))
45+
"or run `pyms crypt create-key` command".format(self._loader.path, crypt_file_key_env))
4246
return key
4347

4448
def encrypt(self, message):
@@ -57,3 +61,10 @@ def decrypt(self, encrypted):
5761

5862
def delete_key(self):
5963
os.remove(self._loader.get_path_from_env())
64+
65+
@staticmethod
66+
def __get_updated_crypt_file_key_env() -> str:
67+
result = CRYPT_FILE_KEY_ENVIRONMENT
68+
if (os.getenv(CRYPT_FILE_KEY_ENVIRONMENT_LEGACY) is not None) and (os.getenv(CRYPT_FILE_KEY_ENVIRONMENT) is None):
69+
result = CRYPT_FILE_KEY_ENVIRONMENT_LEGACY
70+
return result

‎pyms/flask/app/create_app.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ def example():
3939
app.run()
4040
```
4141
Environments variables of PyMS:
42-
**CONFIGMAP_FILE**: The path to the configuration file. By default, PyMS search the configuration file in your
42+
**PYMS_CONFIGMAP_FILE**: The path to the configuration file. By default, PyMS search the configuration file in your
4343
actual folder with the name "config.yml"
4444
4545
## Create configuration

‎pyms/utils/utils.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,19 @@
55
from pyms.exceptions import PackageNotExists
66

77

8+
class Colors:
9+
BLUE = "\033[34m"
10+
GREEN = "\033[32m"
11+
MAGENTA = "\033[35m"
12+
RED = "\033[31m"
13+
YELLOW = "\033[33m"
14+
BRIGHT_BLUE = "\033[94m"
15+
BRIGHT_GREEN = "\033[92m"
16+
BRIGHT_MAGENTA = "\033[95m"
17+
BRIGHT_RED = "\033[91m"
18+
BRIGHT_YELLOW = "\033[93m"
19+
20+
821
def import_from(module: Text, name: Text):
922
module = __import__(module, fromlist=[name])
1023
return getattr(module, name)
@@ -20,3 +33,10 @@ def check_package_exists(package_name: Text) -> Union[Exception, bool]:
2033
raise PackageNotExists(
2134
"{package} is not installed. try with pip install -U {package}".format(package=package_name))
2235
return True
36+
37+
38+
def colored_text(msg, color: Colors, bold=False):
39+
result = "{}{}{}".format(color, msg, "\033[0m")
40+
if bold:
41+
result = "{}{}".format("\033[1m", result)
42+
return result

‎tests/config-tests-debug-off.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
pyms:
2+
services:
3+
metrics: true
4+
requests:
5+
data: data
6+
swagger:
7+
path: ""
8+
file: "swagger.yaml"
9+
tracer:
10+
client: "jaeger"
11+
host: "localhost"
12+
component_name: "Python Microservice"
13+
config:
14+
DEBUG: false
15+
testing: true
16+
app_name: "Python Microservice"
17+
test_var: general
18+
subservice1:
19+
test: input
20+
subservice2:
21+
test: output

‎tests/config-tests-debug.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
pyms:
2+
services:
3+
metrics: true
4+
requests:
5+
data: data
6+
swagger:
7+
path: ""
8+
file: "swagger.yaml"
9+
tracer:
10+
client: "jaeger"
11+
host: "localhost"
12+
component_name: "Python Microservice"
13+
config:
14+
DEBUG: true
15+
testing: true
16+
app_name: "Python Microservice"
17+
test_var: general
18+
subservice1:
19+
test: input
20+
subservice2:
21+
test: output

‎tests/config-tests-deprecated.yml

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
pyms:
2+
services:
3+
metrics: true
4+
requests:
5+
data: data
6+
swagger:
7+
path: ""
8+
file: "swagger.yaml"
9+
tracer:
10+
client: "jaeger"
11+
host: "localhost"
12+
component_name: "Python Microservice"
13+
config:
14+
DEBUG: false
15+
testing: true
16+
app_name: "Python Microservice"
17+
test_var: deprecated
18+
subservice1:
19+
test: input
20+
subservice2:
21+
test: output

‎tests/test_cmd.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ def test_crypt_file_error(self):
2121
cmd = Command(arguments=arguments, autorun=False)
2222
with pytest.raises(FileDoesNotExistException) as excinfo:
2323
cmd.run()
24-
assert ("Decrypt key None not exists. You must set a correct env var KEY_FILE or run "
24+
assert ("Decrypt key None not exists. You must set a correct env var PYMS_KEY_FILE or run "
2525
"`pyms crypt create-key` command") \
2626
in str(excinfo.value)
2727

0 commit comments

Comments
 (0)
0