{

"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "g_a9QvUFVCUR"
},
"source": [
"<h1>Chapter 11 - Fine-tuning Representation Models for Classification</h1>\n",
"<i>Exploring the performance in classification of representation models.</i>\
n",
"\n",
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/
dp/1098150961\"><img src=\"https://img.shields.io/badge/Buy%20the%20Book!-grey?
logo=amazon\"></a>\n",
"<a href=\"https://www.oreilly.com/library/view/hands-on-large-language/
9781098150952/\"><img src=\"https://img.shields.io/badge/O'Reilly-white.svg?
logo=data:image/svg
%2bxml;base64,PHN2ZyB3aWR0aD0iMzQiIGhlaWdodD0iMjciIHZpZXdCb3g9IjAgMCAzNCAyNyIgZmlsb
D0ibm9uZSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KPGNpcmNsZSBjeD0iMTMiIGN5
PSIxNCIgcj0iMTEiIHN0cm9rZT0iI0Q0MDEwMSIgc3Ryb2tlLXdpZHRoPSI0Ii8+CjxjaXJjbGUgY3g9IjM
wLjUiIGN5PSIzLjUiIHI9IjMuNSIgZmlsbD0iI0Q0MDEwMSIvPgo8L3N2Zz4K\"></a>\n",
"<a href=\"https://github.com/HandsOnLLM/Hands-On-Large-Language-Models\"><img
src=\"https://img.shields.io/badge/GitHub%20Repository-black?logo=github\"></a>\n",
"[]
(https://colab.research.google.com/github/HandsOnLLM/Hands-On-Large-Language-
Models/blob/main/chapter11/Chapter%2011%20-%20Fine-Tuning%20BERT.ipynb)\n",
"\n",
"---\n",
"\n",
"This notebook is for Chapter 11 of the [Hands-On Large Language Models]
(https://www.amazon.com/Hands-Large-Language-Models-Understanding/dp/1098150961)
book by [Jay Alammar](https://www.linkedin.com/in/jalammar) and [Maarten
Grootendorst](https://www.linkedin.com/in/mgrootendorst/).\n",
"\n",
"---\n",
"\n",
"<a href=\"https://www.amazon.com/Hands-Large-Language-Models-Understanding/
dp/1098150961\">\n",
"<img src=\"https://raw.githubusercontent.com/HandsOnLLM/Hands-On-Large-
Language-Models/main/images/book_cover.png\" width=\"350\"/></a>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### [OPTIONAL] - Installing Packages on <img
src=\"https://colab.google/static/images/icons/colab.png\" width=100>\n",
"\n",
"If you are viewing this notebook on Google Colab (or any other cloud vendor),
you need to **uncomment and run** the following codeblock to install the
dependencies for this chapter:\n",
"\n",
"---\n",
"\n",
"💡 **NOTE**: We will want to use a GPU to run the examples in this notebook. In
Google Colab, go to\n",
 "**Runtime > Change runtime type > Hardware accelerator > GPU > GPU type >
T4**.\n",
"\n",
"---\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# %%capture\n",
"# !pip install \"datasets>=2.18.0,<3\" transformers>=4.38.2 sentence-
transformers>=2.5.1 setfit>=1.0.3 accelerate>=0.27.2 seqeval>=1.2.2"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "UBeVnXxQWy7-"
},
"source": [
"## **Data**"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 365,
"referenced_widgets": [
"e3c168a65bd34a64b2c4d13129f9d750",
"5ad968464c934a15b19e3924663e5ebf",
"31756f672e60432da8aa12b033a2ea20",
"63adb81b029744fb82e31b3f1190bb9b",
"77511fda45a940d1a6664438b07712b2",
"677310bbba174f3a828d7f625a483f92",
"bbb27e92d9e848e9ada7ba0c9866f87a",
"d6299b347ade47bba93df3eff231be6f",
"49541355ed224e748754ed4bd89792d8",
"7cc8919e7f9e4db48a876619edff0952",
"fdb9b32fdccf4156b537df8e000ec7d2",
"0cd7314417ab48f99adf352f0cab863f",
"bbb68b7249674351abfa8661b4645e2e",
"285d6e493dcb462598190862b186b17e",
"29419f20fad94db88b6da3b1ecd04725",
"3d58f1665e0949d1baa1feaffd52aa29",
"82c561fe82084baaa9eedfb78ec476da",
"10165d819bea43f1ad531ee0a314d002",
"ed2ca51dff8847648010e7b28f03d41e",
"9f216034e74a4f0a9794a812addc82c5",
"d3748aec876a4ddbb377f16a43b3c23e",
"081b5fcad38b4627b1d553244a1bed0c",
"3a4ca5b37d6f426a9d294ade901f6984",
"673f06ed6138437680a01409cbce3a13",
"625db697c4084a4fa7f78905c5c830b2",
 "a6c837a7a4fb45bcb035537cf41796e3",
"0f53e4afed4a42a5a63d8a64f847c27e",
"127978e95632464fb3b7396de66ff240",
"a7fa1b4de3b6423783861a6d8213f430",
"c185b6d9a8514c71b36918027a424204",
"9fc58f20135e48b897cb6a2b8a67863d",
"24b94b5fc6fd4aa7ae113d9eeb3a4d6f",
"7c386e049403493997772cf4aca4a234",
"09a021dba22548f3bd101e237c5ce51d",
"36fdfe575de440458c4fe7cd572d2a22",
"7ef0c97a8b774e57a041a210c3e23ce6",
"d9379ba58d8e4de9b4ba3cc12be87b4c",
"ee301aff657e405c8cdfd2e0f5d7d19e",
"87d2c368a3d44c1994c8fe31fea59212",
"0db4bd167da44ee7b9fc044d9bb5834d",
"ebc2ddfef4cd4129997138c615cd15a1",
"1b0fa0592da147fcab20f0c17f487542",
"0a5befced7554f0a92523b5dca953ed4",
"ec515b2a14ff4414baddca0cc672ef14",
"2eaded0fbbd74c67b802fbb92973fa3a",
"d1e9c32d66004c70b0e3fa96ccb1e4e3",
"d27a323324cd45789808c22917fa5d36",
"7d827c0ee22b47c4b46fd4f59ac3ec23",
"aba270b810d6461887633f62cf35195c",
"f43737300b254f2bb3aef90fabf652e2",
"ff95c667621d46b89412c0638ce91673",
"5f810926256a4a3aa5271c504cdad278",
"117e9997a5db42e59e1f73eddd345b25",
"c807a7269c2a4f8daf8022a177175213",
"c123b5dae69741d6a10b0b783f5e54ac",
"5a5d72b8d2484c54977e5ec692a00004",
"fd5ff63e2c204836b1c6924633880077",
"1c96b599ae6d466e94062f21e7d3a54b",
"690d9fa381b74085af8025af1dcffdcf",
"95ab40e005a9419aa7827dbcbd8a2415",
"aeffa7363e144c0a9b830c7b532167cb",
"59cd08daef304e78be3574b4345084b2",
"1700973179db4c91a9c00fe5e84de1a9",
"5a22333fd16d46429e16f8b1ed9276a1",
"34e7d842cf364b9f8b377b17befa20b2",
"84454ba3262b48d489215b874bd78842",
"91552574464449e48e54a6939f0a5d18",
"0f4c3ac40bf84e6293419447a7fbdd15",
"c82ed37a975149d8895adc331aae34eb",
"96252dca0e9049fea03228e7ceba066b",
"25ed22abc1934a3393593d5b10331988",
"b883ce44ce9e47d6b5ac70e36e049559",
"17afecdfb20b4e49a79aea97d20a10dd",
"c6b1edc0dece4354b7a43314c419cb71",
"19b3c0979b3f45499faa57f341d6d53a",
"ef4d27b4292240928439104f38c610b9",
"ce07f53ddea3469282b20581a4f4e4da"
]
},
"executionInfo": {
"elapsed": 21399,
"status": "ok",
"timestamp": 1719386835780,
"user": {
 "displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "5phRS_z2U_3T",
"outputId": "439a9900-6d9a-4a53-ea30-4774d3bf4de7"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89:
UserWarning: \n",
"The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
"To authenticate with the Hugging Face Hub, create a token in your settings
tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab
and restart your session.\n",
"You will be able to reuse this secret in all of your notebooks.\n",
"Please note that authentication is recommended but still optional to access
public models or datasets.\n",
" warnings.warn(\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e3c168a65bd34a64b2c4d13129f9d750",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading readme: 0%| | 0.00/7.46k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0cd7314417ab48f99adf352f0cab863f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading data: 0%| | 0.00/699k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3a4ca5b37d6f426a9d294ade901f6984",
"version_major": 2,
"version_minor": 0
 },
"text/plain": [
"Downloading data: 0%| | 0.00/90.0k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "09a021dba22548f3bd101e237c5ce51d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading data: 0%| | 0.00/92.2k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2eaded0fbbd74c67b802fbb92973fa3a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating train split: 0%| | 0/8530 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5a5d72b8d2484c54977e5ec692a00004",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating validation split: 0%| | 0/1066 [00:00<?, ?
examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "91552574464449e48e54a6939f0a5d18",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
 "Generating test split: 0%| | 0/1066 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from datasets import load_dataset\n",
"\n",
"# Prepare data and splits\n",
"tomatoes = load_dataset(\"rotten_tomatoes\")\n",
"train_data, test_data = tomatoes[\"train\"], tomatoes[\"test\"]"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xya5dfmVoR1R"
},
"source": [
"## **Supervised Classification**"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "ODEmWQe8W_le"
},
"source": [
"### HuggingFace Trainer"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 232,
"referenced_widgets": [
"1d0ff62806e04095a64ea4c9c299a276",
"17bfe3f4e34a4467ac82642d905877ee",
"2dcbebcd7c1a4d9db8783e85609e6419",
"61ae7efc64864dacb03354a9189d0034",
"f6ceedd97cf84a959d2b631bc677bf0c",
"e266572c32be4532ae59b0af652ee0f9",
"b388229af88446358932d199e0fbf70f",
"ee169c0ee96442b0b0acbbd4ef2feedc",
"58e0fb0d5c2b401ab49ce3d6a1a46ba9",
"514f1594aa114a5e98aaa2f3197ff106",
"726177412b3d4c8f9ca9f481fafcb41a",
"4bf958fcb13740d29ad463bd4f9b7e5c",
"7ed87ac491764e4db2d23635a81c4246",
"67da8e11fe7f49589c923326c7e6e19c",
"f26b357136fd48a1acaddc2969147337",
"0c75544ff7784990998c8a398b470f78",
"efae0be4bc7f41a8a23ca5c23f174f9e",
"d401147edaa0404b9f00dc881f895509",
 "e1edb2b21532480eb6df74eafeb876a0",
"a493195172fd48239c720a874acd9d83",
"d993c04399af4bea87e15db60b7bcb4c",
"b3de189dfd9948aa958854707b5bc76f",
"3b9c918bb3f04232a9dac11f4e46baf8",
"bae0792a9ba44c628906dca306d656df",
"f82d95c50ea946a79a1587b472367f99",
"2285d28a20e142169d848e3a3641eec0",
"a7a9040968ac41f69df95da42e853471",
"b61d12afe30d4d4c8bb3808225d6ce3f",
"7e5b26db97454d389794ab03f784f975",
"cb751178c50f43fb8a06c63218ea8029",
"6fa29e1fc9c143269d68a72a3349c363",
"c91875964a24495eb1a0395375a1cc07",
"5b7fdd9526fb4f96a3f178d021fc51ff",
"5c43d75618834ca49ff92c52c725dfdc",
"6d1656f89dbf4af3914ad5b5f7cb8570",
"ad52002217cf4b85b148f1dd870871f3",
"328ab43065d647e58d7bc53913107b9a",
"c2aa35abf89442079f907b9928dda445",
"281bed63cd5447559eb8305d9eda0b06",
"7d5a3b0eba09486f88cd3e857790fa5d",
"3ef14566be734f23a7ccbcc446f94073",
"513dc889d5eb413e872b807cc98f64bb",
"d0344e9d2d914fb29f5ce4f7cceca39c",
"0d45042f22674eb7abcf033a7c34ce3c",
"35498c89148e4e548e110ff979b33dd8",
"fcb89932078f484da12e2916ec62ecc1",
"917ed901f0e74c9eaaa8cedf6b24b1f7",
"8fb9a0880b4740b4b78e125b31959f4b",
"9b5ed41f3ede47bf9917830804dd4958",
"9f0bcc78563a4e7895c8fa9f304b17a9",
"668812cc3d6d45d393437bb299816015",
"a1c09a06df704b1388759228eac0aa69",
"cf599267ff80457ca1080bc7d5636b90",
"224eb4bbc40240b98ca9128066d301c3",
"11dc85e0f99d48a68cec364dcdb7d57d"
]
},
"executionInfo": {
"elapsed": 9171,
"status": "ok",
"timestamp": 1719386844950,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "b-1UGAg7WAHk",
"outputId": "e1fd7d6f-5b19-4fd7-8be0-690afe164820"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "1d0ff62806e04095a64ea4c9c299a276",
"version_major": 2,
"version_minor": 0
 },
"text/plain": [
"config.json: 0%| | 0.00/570 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "4bf958fcb13740d29ad463bd4f9b7e5c",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"model.safetensors: 0%| | 0.00/436M [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertForSequenceClassification were not initialized from the
model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias',
'classifier.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use
it for predictions and inference.\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3b9c918bb3f04232a9dac11f4e46baf8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer_config.json: 0%| | 0.00/49.0 [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5c43d75618834ca49ff92c52c725dfdc",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"vocab.txt: 0%| | 0.00/213k [00:00<?, ?B/s]"
]
},
"metadata": {},
 "output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "35498c89148e4e548e110ff979b33dd8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"tokenizer.json: 0%| | 0.00/436k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from transformers import AutoTokenizer, AutoModelForSequenceClassification\n",
"\n",
"# Load Model and Tokenizer\n",
"model_id = \"bert-base-cased\"\n",
"model = AutoModelForSequenceClassification.from_pretrained(model_id,
num_labels=2)\n",
"tokenizer = AutoTokenizer.from_pretrained(model_id)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "r48vDo8fa33D"
},
"source": [
"Tokenize our data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81,
"referenced_widgets": [
"a3ee82becf164af19d6ad09d7897a306",
"bd64c7b50ad345a9b9f1c253fff11b9a",
"e4038ea26e7248b08a9bf77390ff7cf5",
"4ff9a26f1e274b47883cedbf09f40c20",
"76dcdbcd894640fe92b894bb9497ca78",
"c94e820825704d9fb42b6ab5b56c8a58",
"ca8c5e06e4404c39a944f68ecdd64eb2",
"e9087132832c4b64b4226d9348c3e3fe",
"49a0d9996c5a4dacb7be81b0c79447a7",
"7b30340529994e2bb6c1584009eaa165",
"52dbbd4610d54640a1ce46c767cb79cd",
"a5637ae2cbd44f21aee434d399e2be37",
"9bdb983954064b50ab80865c26a74edd",
"2b272649b2844a98b4d66c8e4dc32798",
"71ab8ce53934408d94cf07d60c845380",
 "a46b81d1b5384fd8b11a9404864b56c8",
"928473597d704fc9bc95c9b00c5185ec",
"86fcfb7fa0b742b3b1afad960d084ab3",
"96249184adf84fcbbfebf5872aab29a2",
"399cd589edfd4db2ac8ecc29283c114f",
"482bcf30a8204361b610a7e04627e57e",
"65dc3e08e34b463385d782600a7096b3"
]
},
"executionInfo": {
"elapsed": 5163,
"status": "ok",
"timestamp": 1719386850111,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "5ySNm-a3WCFI",
"outputId": "892e96a5-7161-485c-b514-9e24ec117394"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a3ee82becf164af19d6ad09d7897a306",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/8530 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a5637ae2cbd44f21aee434d399e2be37",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/1066 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from transformers import DataCollatorWithPadding\n",
"\n",
"# Pad to the longest sequence in the batch\n",
"data_collator = DataCollatorWithPadding(tokenizer=tokenizer)\n",
"\n",
"def preprocess_function(examples):\n",
 " \"\"\"Tokenize input data\"\"\"\n",
" return tokenizer(examples[\"text\"], truncation=True)\n",
"\n",
"# Tokenize train/test data\n",
"tokenized_train = train_data.map(preprocess_function, batched=True)\n",
"tokenized_test = test_data.map(preprocess_function, batched=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QJ1NM1gjbMD7"
},
"source": [
"Define metrics."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Y724gUYyWIvq"
},
"outputs": [],
"source": [
"import numpy as np\n",
"import evaluate\n",
"\n",
"\n",
"def compute_metrics(eval_pred):\n",
" \"\"\"Calculate F1 score\"\"\"\n",
" logits, labels = eval_pred\n",
" predictions = np.argmax(logits, axis=-1)\n",
"\n",
" load_f1 = evaluate.load(\"f1\")\n",
" f1 = load_f1.compute(predictions=predictions, references=labels)[\"f1\"]\
n",
" return {\"f1\": f1}"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AAm-sAl9bOPC"
},
"source": [
"Train model."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Dho6VcG9WK5u"
},
"outputs": [],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"\n",
 "# Training arguments for parameter tuning\n",
"training_args = TrainingArguments(\n",
" \"model\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=16,\n",
" per_device_eval_batch_size=16,\n",
" num_train_epochs=1,\n",
" weight_decay=0.01,\n",
" save_strategy=\"epoch\",\n",
" report_to=\"none\"\n",
")\n",
"\n",
"# Trainer which executes the training process\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_train,\n",
" eval_dataset=tokenized_test,\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"executionInfo": {
"elapsed": 62039,
"status": "ok",
"timestamp": 1719386995069,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "vOzl0WnSbVnY",
"outputId": "0058b56f-6a7c-40d8-f08f-2bd035d7ab79"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='534' max='534' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [534/534 01:00, Epoch 1/1]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
 " <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>500</td>\n",
" <td>0.424000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=534, training_loss=0.4183677966228585,
metrics={'train_runtime': 61.6658, 'train_samples_per_second': 138.326,
'train_steps_per_second': 8.66, 'total_flos': 227605451772240.0, 'train_loss':
0.4183677966228585, 'epoch': 1.0})"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.train()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "xkBUVlUYbUnn"
},
"source": [
"Evaluate results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 173,
"referenced_widgets": [
"d656514d509441548af1dd6c7326d18a",
"05ce4f6b47354c628c0a41544b9419f5",
"d55ba50d84d7412daeb6ccc73be98694",
"d5d54eddd21a4999aec6ea6a7782e223",
"e8da2d90a93b4ce9a93c2ee93f75d59b",
 "b94e1b170ccc4ff7abbb9ed87ad2aedd",
"3c38333cafb446bab3f3f75824075927",
"960641d3e97f4a538a135861ab058bca",
"fc4d3609cada4615924d5e92ee1ffdf4",
"f914214414b14945a72ac57ac93dcb5e",
"8cb0531c591c43f098564ce043eb036f"
]
},
"executionInfo": {
"elapsed": 3959,
"status": "ok",
"timestamp": 1719386999026,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "wCI9uYDObWU8",
"outputId": "5c2d54d2-0c81-4a10-d46d-9d7c4c9405d3"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='67' max='67' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [67/67 00:01]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d656514d509441548af1dd6c7326d18a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading builder script: 0%| | 0.00/6.77k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.37090229988098145,\n",
 " 'eval_f1': 0.8566073102155576,\n",
" 'eval_runtime': 3.1133,\n",
" 'eval_samples_per_second': 342.407,\n",
" 'eval_steps_per_second': 21.521,\n",
" 'epoch': 1.0}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.evaluate()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "o5gefwxOBllA"
},
"source": [
"### Freeze Layers"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 6,
"status": "ok",
"timestamp": 1719386999026,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "t0ZOuoe7Dj3c",
"outputId": "00b9fb9c-f819-457d-91ad-a57881778d1e"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertForSequenceClassification were not initialized from the
model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias',
'classifier.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use
it for predictions and inference.\n"
]
}
],
"source": [
 "# Load Model and Tokenizer\n",
"model = AutoModelForSequenceClassification.from_pretrained(model_id,
num_labels=2)\n",
"tokenizer = AutoTokenizer.from_pretrained(model_id)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 5,
"status": "ok",
"timestamp": 1719386999026,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "iI8vf_mnBniu",
"outputId": "d5d31516-977b-462b-f878-5a71c084db7b"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bert.embeddings.word_embeddings.weight\n",
"bert.embeddings.position_embeddings.weight\n",
"bert.embeddings.token_type_embeddings.weight\n",
"bert.embeddings.LayerNorm.weight\n",
"bert.embeddings.LayerNorm.bias\n",
"bert.encoder.layer.0.attention.self.query.weight\n",
"bert.encoder.layer.0.attention.self.query.bias\n",
"bert.encoder.layer.0.attention.self.key.weight\n",
"bert.encoder.layer.0.attention.self.key.bias\n",
"bert.encoder.layer.0.attention.self.value.weight\n",
"bert.encoder.layer.0.attention.self.value.bias\n",
"bert.encoder.layer.0.attention.output.dense.weight\n",
"bert.encoder.layer.0.attention.output.dense.bias\n",
"bert.encoder.layer.0.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.0.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.0.intermediate.dense.weight\n",
"bert.encoder.layer.0.intermediate.dense.bias\n",
"bert.encoder.layer.0.output.dense.weight\n",
"bert.encoder.layer.0.output.dense.bias\n",
"bert.encoder.layer.0.output.LayerNorm.weight\n",
"bert.encoder.layer.0.output.LayerNorm.bias\n",
"bert.encoder.layer.1.attention.self.query.weight\n",
"bert.encoder.layer.1.attention.self.query.bias\n",
"bert.encoder.layer.1.attention.self.key.weight\n",
"bert.encoder.layer.1.attention.self.key.bias\n",
"bert.encoder.layer.1.attention.self.value.weight\n",
"bert.encoder.layer.1.attention.self.value.bias\n",
"bert.encoder.layer.1.attention.output.dense.weight\n",
"bert.encoder.layer.1.attention.output.dense.bias\n",
"bert.encoder.layer.1.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.1.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.1.intermediate.dense.weight\n",
"bert.encoder.layer.1.intermediate.dense.bias\n",
"bert.encoder.layer.1.output.dense.weight\n",
"bert.encoder.layer.1.output.dense.bias\n",
"bert.encoder.layer.1.output.LayerNorm.weight\n",
"bert.encoder.layer.1.output.LayerNorm.bias\n",
"bert.encoder.layer.2.attention.self.query.weight\n",
"bert.encoder.layer.2.attention.self.query.bias\n",
"bert.encoder.layer.2.attention.self.key.weight\n",
"bert.encoder.layer.2.attention.self.key.bias\n",
"bert.encoder.layer.2.attention.self.value.weight\n",
"bert.encoder.layer.2.attention.self.value.bias\n",
"bert.encoder.layer.2.attention.output.dense.weight\n",
"bert.encoder.layer.2.attention.output.dense.bias\n",
"bert.encoder.layer.2.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.2.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.2.intermediate.dense.weight\n",
"bert.encoder.layer.2.intermediate.dense.bias\n",
"bert.encoder.layer.2.output.dense.weight\n",
"bert.encoder.layer.2.output.dense.bias\n",
"bert.encoder.layer.2.output.LayerNorm.weight\n",
"bert.encoder.layer.2.output.LayerNorm.bias\n",
"bert.encoder.layer.3.attention.self.query.weight\n",
"bert.encoder.layer.3.attention.self.query.bias\n",
"bert.encoder.layer.3.attention.self.key.weight\n",
"bert.encoder.layer.3.attention.self.key.bias\n",
"bert.encoder.layer.3.attention.self.value.weight\n",
"bert.encoder.layer.3.attention.self.value.bias\n",
"bert.encoder.layer.3.attention.output.dense.weight\n",
"bert.encoder.layer.3.attention.output.dense.bias\n",
"bert.encoder.layer.3.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.3.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.3.intermediate.dense.weight\n",
"bert.encoder.layer.3.intermediate.dense.bias\n",
"bert.encoder.layer.3.output.dense.weight\n",
"bert.encoder.layer.3.output.dense.bias\n",
"bert.encoder.layer.3.output.LayerNorm.weight\n",
"bert.encoder.layer.3.output.LayerNorm.bias\n",
"bert.encoder.layer.4.attention.self.query.weight\n",
"bert.encoder.layer.4.attention.self.query.bias\n",
"bert.encoder.layer.4.attention.self.key.weight\n",
"bert.encoder.layer.4.attention.self.key.bias\n",
"bert.encoder.layer.4.attention.self.value.weight\n",
"bert.encoder.layer.4.attention.self.value.bias\n",
"bert.encoder.layer.4.attention.output.dense.weight\n",
"bert.encoder.layer.4.attention.output.dense.bias\n",
"bert.encoder.layer.4.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.4.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.4.intermediate.dense.weight\n",
"bert.encoder.layer.4.intermediate.dense.bias\n",
"bert.encoder.layer.4.output.dense.weight\n",
"bert.encoder.layer.4.output.dense.bias\n",
"bert.encoder.layer.4.output.LayerNorm.weight\n",
"bert.encoder.layer.4.output.LayerNorm.bias\n",
"bert.encoder.layer.5.attention.self.query.weight\n",
"bert.encoder.layer.5.attention.self.query.bias\n",
"bert.encoder.layer.5.attention.self.key.weight\n",
"bert.encoder.layer.5.attention.self.key.bias\n",
"bert.encoder.layer.5.attention.self.value.weight\n",
"bert.encoder.layer.5.attention.self.value.bias\n",
"bert.encoder.layer.5.attention.output.dense.weight\n",
"bert.encoder.layer.5.attention.output.dense.bias\n",
"bert.encoder.layer.5.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.5.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.5.intermediate.dense.weight\n",
"bert.encoder.layer.5.intermediate.dense.bias\n",
"bert.encoder.layer.5.output.dense.weight\n",
"bert.encoder.layer.5.output.dense.bias\n",
"bert.encoder.layer.5.output.LayerNorm.weight\n",
"bert.encoder.layer.5.output.LayerNorm.bias\n",
"bert.encoder.layer.6.attention.self.query.weight\n",
"bert.encoder.layer.6.attention.self.query.bias\n",
"bert.encoder.layer.6.attention.self.key.weight\n",
"bert.encoder.layer.6.attention.self.key.bias\n",
"bert.encoder.layer.6.attention.self.value.weight\n",
"bert.encoder.layer.6.attention.self.value.bias\n",
"bert.encoder.layer.6.attention.output.dense.weight\n",
"bert.encoder.layer.6.attention.output.dense.bias\n",
"bert.encoder.layer.6.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.6.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.6.intermediate.dense.weight\n",
"bert.encoder.layer.6.intermediate.dense.bias\n",
"bert.encoder.layer.6.output.dense.weight\n",
"bert.encoder.layer.6.output.dense.bias\n",
"bert.encoder.layer.6.output.LayerNorm.weight\n",
"bert.encoder.layer.6.output.LayerNorm.bias\n",
"bert.encoder.layer.7.attention.self.query.weight\n",
"bert.encoder.layer.7.attention.self.query.bias\n",
"bert.encoder.layer.7.attention.self.key.weight\n",
"bert.encoder.layer.7.attention.self.key.bias\n",
"bert.encoder.layer.7.attention.self.value.weight\n",
"bert.encoder.layer.7.attention.self.value.bias\n",
"bert.encoder.layer.7.attention.output.dense.weight\n",
"bert.encoder.layer.7.attention.output.dense.bias\n",
"bert.encoder.layer.7.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.7.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.7.intermediate.dense.weight\n",
"bert.encoder.layer.7.intermediate.dense.bias\n",
"bert.encoder.layer.7.output.dense.weight\n",
"bert.encoder.layer.7.output.dense.bias\n",
"bert.encoder.layer.7.output.LayerNorm.weight\n",
"bert.encoder.layer.7.output.LayerNorm.bias\n",
"bert.encoder.layer.8.attention.self.query.weight\n",
"bert.encoder.layer.8.attention.self.query.bias\n",
"bert.encoder.layer.8.attention.self.key.weight\n",
"bert.encoder.layer.8.attention.self.key.bias\n",
"bert.encoder.layer.8.attention.self.value.weight\n",
"bert.encoder.layer.8.attention.self.value.bias\n",
"bert.encoder.layer.8.attention.output.dense.weight\n",
"bert.encoder.layer.8.attention.output.dense.bias\n",
"bert.encoder.layer.8.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.8.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.8.intermediate.dense.weight\n",
"bert.encoder.layer.8.intermediate.dense.bias\n",
"bert.encoder.layer.8.output.dense.weight\n",
 "bert.encoder.layer.8.output.dense.bias\n",
"bert.encoder.layer.8.output.LayerNorm.weight\n",
"bert.encoder.layer.8.output.LayerNorm.bias\n",
"bert.encoder.layer.9.attention.self.query.weight\n",
"bert.encoder.layer.9.attention.self.query.bias\n",
"bert.encoder.layer.9.attention.self.key.weight\n",
"bert.encoder.layer.9.attention.self.key.bias\n",
"bert.encoder.layer.9.attention.self.value.weight\n",
"bert.encoder.layer.9.attention.self.value.bias\n",
"bert.encoder.layer.9.attention.output.dense.weight\n",
"bert.encoder.layer.9.attention.output.dense.bias\n",
"bert.encoder.layer.9.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.9.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.9.intermediate.dense.weight\n",
"bert.encoder.layer.9.intermediate.dense.bias\n",
"bert.encoder.layer.9.output.dense.weight\n",
"bert.encoder.layer.9.output.dense.bias\n",
"bert.encoder.layer.9.output.LayerNorm.weight\n",
"bert.encoder.layer.9.output.LayerNorm.bias\n",
"bert.encoder.layer.10.attention.self.query.weight\n",
"bert.encoder.layer.10.attention.self.query.bias\n",
"bert.encoder.layer.10.attention.self.key.weight\n",
"bert.encoder.layer.10.attention.self.key.bias\n",
"bert.encoder.layer.10.attention.self.value.weight\n",
"bert.encoder.layer.10.attention.self.value.bias\n",
"bert.encoder.layer.10.attention.output.dense.weight\n",
"bert.encoder.layer.10.attention.output.dense.bias\n",
"bert.encoder.layer.10.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.10.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.10.intermediate.dense.weight\n",
"bert.encoder.layer.10.intermediate.dense.bias\n",
"bert.encoder.layer.10.output.dense.weight\n",
"bert.encoder.layer.10.output.dense.bias\n",
"bert.encoder.layer.10.output.LayerNorm.weight\n",
"bert.encoder.layer.10.output.LayerNorm.bias\n",
"bert.encoder.layer.11.attention.self.query.weight\n",
"bert.encoder.layer.11.attention.self.query.bias\n",
"bert.encoder.layer.11.attention.self.key.weight\n",
"bert.encoder.layer.11.attention.self.key.bias\n",
"bert.encoder.layer.11.attention.self.value.weight\n",
"bert.encoder.layer.11.attention.self.value.bias\n",
"bert.encoder.layer.11.attention.output.dense.weight\n",
"bert.encoder.layer.11.attention.output.dense.bias\n",
"bert.encoder.layer.11.attention.output.LayerNorm.weight\n",
"bert.encoder.layer.11.attention.output.LayerNorm.bias\n",
"bert.encoder.layer.11.intermediate.dense.weight\n",
"bert.encoder.layer.11.intermediate.dense.bias\n",
"bert.encoder.layer.11.output.dense.weight\n",
"bert.encoder.layer.11.output.dense.bias\n",
"bert.encoder.layer.11.output.LayerNorm.weight\n",
"bert.encoder.layer.11.output.LayerNorm.bias\n",
"bert.pooler.dense.weight\n",
"bert.pooler.dense.bias\n",
"classifier.weight\n",
"classifier.bias\n"
]
}
],
"source": [
 "# Print layer names\n",
"for name, param in model.named_parameters():\n",
" print(name)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FnpGOry_Bm36"
},
"outputs": [],
"source": [
"for name, param in model.named_parameters():\n",
"\n",
" # Trainable classification head\n",
" if name.startswith(\"classifier\"):\n",
" param.requires_grad = True\n",
"\n",
" # Freeze everything else\n",
" else:\n",
" param.requires_grad = False"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 4,
"status": "ok",
"timestamp": 1719386999026,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "vf_wYzpMB4uX",
"outputId": "8a63f6d8-4516-4d84-d0d5-39f1823b4d03"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Parameter: bert.embeddings.word_embeddings.weight ----- False\n",
"Parameter: bert.embeddings.position_embeddings.weight ----- False\n",
"Parameter: bert.embeddings.token_type_embeddings.weight ----- False\n",
"Parameter: bert.embeddings.LayerNorm.weight ----- False\n",
"Parameter: bert.embeddings.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.0.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.0.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.0.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.0.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.0.attention.self.value.weight ----- False\n",
 "Parameter: bert.encoder.layer.0.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.0.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.0.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.0.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.0.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.0.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.0.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.0.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.0.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.0.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.0.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.1.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.1.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.1.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.1.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.1.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.1.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.1.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.1.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.1.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.1.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.1.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.2.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.2.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.2.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.2.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.2.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.2.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.2.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.2.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.2.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.2.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.2.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.3.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.3.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.3.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.3.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.3.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.3.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.3.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.3.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.3.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.3.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.3.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.4.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.4.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.4.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.4.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.4.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.4.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.4.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.4.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.4.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.4.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.4.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.5.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.5.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.5.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.5.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.5.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.5.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.5.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.5.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.5.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.5.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.5.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.6.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.6.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.6.attention.output.dense.bias ----- False\n",
 "Parameter: bert.encoder.layer.6.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.6.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.6.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.6.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.6.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.6.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.6.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.6.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.7.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.7.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.7.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.7.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.7.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.7.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.7.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.7.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.7.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.7.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.7.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.8.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.8.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.8.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.8.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.8.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.8.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.8.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.8.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.8.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.8.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.8.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.9.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.9.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.9.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.9.attention.output.LayerNorm.weight -----
False\n",
 "Parameter: bert.encoder.layer.9.attention.output.LayerNorm.bias ----- False\
n",
"Parameter: bert.encoder.layer.9.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.9.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.9.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.9.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.9.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.9.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.10.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.10.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.10.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.10.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.10.attention.output.LayerNorm.bias -----
False\n",
"Parameter: bert.encoder.layer.10.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.10.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.10.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.10.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.10.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.10.output.LayerNorm.bias ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.query.weight ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.query.bias ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.key.weight ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.key.bias ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.value.weight ----- False\n",
"Parameter: bert.encoder.layer.11.attention.self.value.bias ----- False\n",
"Parameter: bert.encoder.layer.11.attention.output.dense.weight ----- False\
n",
"Parameter: bert.encoder.layer.11.attention.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.11.attention.output.LayerNorm.weight -----
False\n",
"Parameter: bert.encoder.layer.11.attention.output.LayerNorm.bias -----
False\n",
"Parameter: bert.encoder.layer.11.intermediate.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.11.intermediate.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.11.output.dense.weight ----- False\n",
"Parameter: bert.encoder.layer.11.output.dense.bias ----- False\n",
"Parameter: bert.encoder.layer.11.output.LayerNorm.weight ----- False\n",
"Parameter: bert.encoder.layer.11.output.LayerNorm.bias ----- False\n",
"Parameter: bert.pooler.dense.weight ----- False\n",
"Parameter: bert.pooler.dense.bias ----- False\n",
"Parameter: classifier.weight ----- True\n",
"Parameter: classifier.bias ----- True\n"
]
}
],
"source": [
"# We can check whether the model was correctly updated\n",
"for name, param in model.named_parameters():\n",
" print(f\"Parameter: {name} ----- {param.requires_grad}\")"
]
},
 {
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"executionInfo": {
"elapsed": 15480,
"status": "ok",
"timestamp": 1719387014504,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "rVi36FJSG4ue",
"outputId": "c2e92d2d-247e-4065-8d06-688e24f472f7"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='534' max='534' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [534/534 00:15, Epoch 1/1]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>500</td>\n",
" <td>0.697000</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=534, training_loss=0.6962381677234664,
metrics={'train_runtime': 15.234, 'train_samples_per_second': 559.931,
'train_steps_per_second': 35.053, 'total_flos': 227605451772240.0, 'train_loss':
0.6962381677234664, 'epoch': 1.0})"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import TrainingArguments, Trainer\n",
"\n",
"# Trainer which executes the training process\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_train,\n",
" eval_dataset=tokenized_test,\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"executionInfo": {
"elapsed": 2623,
"status": "ok",
"timestamp": 1719387017125,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "eCPpixB1HCsI",
"outputId": "3d77ed38-0565-492b-b488-09eb525fc316"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='67' max='67' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [67/67 00:01]\n",
" </div>\n",
" "
],
 "text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.6823198795318604,\n",
" 'eval_f1': 0.637704918032787,\n",
" 'eval_runtime': 2.7203,\n",
" 'eval_samples_per_second': 391.865,\n",
" 'eval_steps_per_second': 24.629,\n",
" 'epoch': 1.0}"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.evaluate()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Uw729mLhIQL6"
},
"source": [
"### Freeze blocks 1-5"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 640,
"status": "ok",
"timestamp": 1719387017762,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "RbsLR561Kje-",
"outputId": "ae1b3e9b-443d-4a06-94bb-6792d1751e27"
},
"outputs": [
{
"name": "stdout",
 "output_type": "stream",
"text": [
"Parameter: 0bert.embeddings.word_embeddings.weight ----- False\n",
"Parameter: 1bert.embeddings.position_embeddings.weight ----- False\n",
"Parameter: 2bert.embeddings.token_type_embeddings.weight ----- False\n",
"Parameter: 3bert.embeddings.LayerNorm.weight ----- False\n",
"Parameter: 4bert.embeddings.LayerNorm.bias ----- False\n",
"Parameter: 5bert.encoder.layer.0.attention.self.query.weight ----- False\n",
"Parameter: 6bert.encoder.layer.0.attention.self.query.bias ----- False\n",
"Parameter: 7bert.encoder.layer.0.attention.self.key.weight ----- False\n",
"Parameter: 8bert.encoder.layer.0.attention.self.key.bias ----- False\n",
"Parameter: 9bert.encoder.layer.0.attention.self.value.weight ----- False\n",
"Parameter: 10bert.encoder.layer.0.attention.self.value.bias ----- False\n",
"Parameter: 11bert.encoder.layer.0.attention.output.dense.weight ----- False\
n",
"Parameter: 12bert.encoder.layer.0.attention.output.dense.bias ----- False\
n",
"Parameter: 13bert.encoder.layer.0.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 14bert.encoder.layer.0.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 15bert.encoder.layer.0.intermediate.dense.weight ----- False\n",
"Parameter: 16bert.encoder.layer.0.intermediate.dense.bias ----- False\n",
"Parameter: 17bert.encoder.layer.0.output.dense.weight ----- False\n",
"Parameter: 18bert.encoder.layer.0.output.dense.bias ----- False\n",
"Parameter: 19bert.encoder.layer.0.output.LayerNorm.weight ----- False\n",
"Parameter: 20bert.encoder.layer.0.output.LayerNorm.bias ----- False\n",
"Parameter: 21bert.encoder.layer.1.attention.self.query.weight ----- False\
n",
"Parameter: 22bert.encoder.layer.1.attention.self.query.bias ----- False\n",
"Parameter: 23bert.encoder.layer.1.attention.self.key.weight ----- False\n",
"Parameter: 24bert.encoder.layer.1.attention.self.key.bias ----- False\n",
"Parameter: 25bert.encoder.layer.1.attention.self.value.weight ----- False\
n",
"Parameter: 26bert.encoder.layer.1.attention.self.value.bias ----- False\n",
"Parameter: 27bert.encoder.layer.1.attention.output.dense.weight ----- False\
n",
"Parameter: 28bert.encoder.layer.1.attention.output.dense.bias ----- False\
n",
"Parameter: 29bert.encoder.layer.1.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 30bert.encoder.layer.1.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 31bert.encoder.layer.1.intermediate.dense.weight ----- False\n",
"Parameter: 32bert.encoder.layer.1.intermediate.dense.bias ----- False\n",
"Parameter: 33bert.encoder.layer.1.output.dense.weight ----- False\n",
"Parameter: 34bert.encoder.layer.1.output.dense.bias ----- False\n",
"Parameter: 35bert.encoder.layer.1.output.LayerNorm.weight ----- False\n",
"Parameter: 36bert.encoder.layer.1.output.LayerNorm.bias ----- False\n",
"Parameter: 37bert.encoder.layer.2.attention.self.query.weight ----- False\
n",
"Parameter: 38bert.encoder.layer.2.attention.self.query.bias ----- False\n",
"Parameter: 39bert.encoder.layer.2.attention.self.key.weight ----- False\n",
"Parameter: 40bert.encoder.layer.2.attention.self.key.bias ----- False\n",
"Parameter: 41bert.encoder.layer.2.attention.self.value.weight ----- False\
n",
"Parameter: 42bert.encoder.layer.2.attention.self.value.bias ----- False\n",
"Parameter: 43bert.encoder.layer.2.attention.output.dense.weight ----- False\
n",
 "Parameter: 44bert.encoder.layer.2.attention.output.dense.bias ----- False\
n",
"Parameter: 45bert.encoder.layer.2.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 46bert.encoder.layer.2.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 47bert.encoder.layer.2.intermediate.dense.weight ----- False\n",
"Parameter: 48bert.encoder.layer.2.intermediate.dense.bias ----- False\n",
"Parameter: 49bert.encoder.layer.2.output.dense.weight ----- False\n",
"Parameter: 50bert.encoder.layer.2.output.dense.bias ----- False\n",
"Parameter: 51bert.encoder.layer.2.output.LayerNorm.weight ----- False\n",
"Parameter: 52bert.encoder.layer.2.output.LayerNorm.bias ----- False\n",
"Parameter: 53bert.encoder.layer.3.attention.self.query.weight ----- False\
n",
"Parameter: 54bert.encoder.layer.3.attention.self.query.bias ----- False\n",
"Parameter: 55bert.encoder.layer.3.attention.self.key.weight ----- False\n",
"Parameter: 56bert.encoder.layer.3.attention.self.key.bias ----- False\n",
"Parameter: 57bert.encoder.layer.3.attention.self.value.weight ----- False\
n",
"Parameter: 58bert.encoder.layer.3.attention.self.value.bias ----- False\n",
"Parameter: 59bert.encoder.layer.3.attention.output.dense.weight ----- False\
n",
"Parameter: 60bert.encoder.layer.3.attention.output.dense.bias ----- False\
n",
"Parameter: 61bert.encoder.layer.3.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 62bert.encoder.layer.3.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 63bert.encoder.layer.3.intermediate.dense.weight ----- False\n",
"Parameter: 64bert.encoder.layer.3.intermediate.dense.bias ----- False\n",
"Parameter: 65bert.encoder.layer.3.output.dense.weight ----- False\n",
"Parameter: 66bert.encoder.layer.3.output.dense.bias ----- False\n",
"Parameter: 67bert.encoder.layer.3.output.LayerNorm.weight ----- False\n",
"Parameter: 68bert.encoder.layer.3.output.LayerNorm.bias ----- False\n",
"Parameter: 69bert.encoder.layer.4.attention.self.query.weight ----- False\
n",
"Parameter: 70bert.encoder.layer.4.attention.self.query.bias ----- False\n",
"Parameter: 71bert.encoder.layer.4.attention.self.key.weight ----- False\n",
"Parameter: 72bert.encoder.layer.4.attention.self.key.bias ----- False\n",
"Parameter: 73bert.encoder.layer.4.attention.self.value.weight ----- False\
n",
"Parameter: 74bert.encoder.layer.4.attention.self.value.bias ----- False\n",
"Parameter: 75bert.encoder.layer.4.attention.output.dense.weight ----- False\
n",
"Parameter: 76bert.encoder.layer.4.attention.output.dense.bias ----- False\
n",
"Parameter: 77bert.encoder.layer.4.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 78bert.encoder.layer.4.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 79bert.encoder.layer.4.intermediate.dense.weight ----- False\n",
"Parameter: 80bert.encoder.layer.4.intermediate.dense.bias ----- False\n",
"Parameter: 81bert.encoder.layer.4.output.dense.weight ----- False\n",
"Parameter: 82bert.encoder.layer.4.output.dense.bias ----- False\n",
"Parameter: 83bert.encoder.layer.4.output.LayerNorm.weight ----- False\n",
"Parameter: 84bert.encoder.layer.4.output.LayerNorm.bias ----- False\n",
"Parameter: 85bert.encoder.layer.5.attention.self.query.weight ----- False\
n",
"Parameter: 86bert.encoder.layer.5.attention.self.query.bias ----- False\n",
 "Parameter: 87bert.encoder.layer.5.attention.self.key.weight ----- False\n",
"Parameter: 88bert.encoder.layer.5.attention.self.key.bias ----- False\n",
"Parameter: 89bert.encoder.layer.5.attention.self.value.weight ----- False\
n",
"Parameter: 90bert.encoder.layer.5.attention.self.value.bias ----- False\n",
"Parameter: 91bert.encoder.layer.5.attention.output.dense.weight ----- False\
n",
"Parameter: 92bert.encoder.layer.5.attention.output.dense.bias ----- False\
n",
"Parameter: 93bert.encoder.layer.5.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 94bert.encoder.layer.5.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 95bert.encoder.layer.5.intermediate.dense.weight ----- False\n",
"Parameter: 96bert.encoder.layer.5.intermediate.dense.bias ----- False\n",
"Parameter: 97bert.encoder.layer.5.output.dense.weight ----- False\n",
"Parameter: 98bert.encoder.layer.5.output.dense.bias ----- False\n",
"Parameter: 99bert.encoder.layer.5.output.LayerNorm.weight ----- False\n",
"Parameter: 100bert.encoder.layer.5.output.LayerNorm.bias ----- False\n",
"Parameter: 101bert.encoder.layer.6.attention.self.query.weight ----- False\
n",
"Parameter: 102bert.encoder.layer.6.attention.self.query.bias ----- False\n",
"Parameter: 103bert.encoder.layer.6.attention.self.key.weight ----- False\n",
"Parameter: 104bert.encoder.layer.6.attention.self.key.bias ----- False\n",
"Parameter: 105bert.encoder.layer.6.attention.self.value.weight ----- False\
n",
"Parameter: 106bert.encoder.layer.6.attention.self.value.bias ----- False\n",
"Parameter: 107bert.encoder.layer.6.attention.output.dense.weight -----
False\n",
"Parameter: 108bert.encoder.layer.6.attention.output.dense.bias ----- False\
n",
"Parameter: 109bert.encoder.layer.6.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 110bert.encoder.layer.6.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 111bert.encoder.layer.6.intermediate.dense.weight ----- False\n",
"Parameter: 112bert.encoder.layer.6.intermediate.dense.bias ----- False\n",
"Parameter: 113bert.encoder.layer.6.output.dense.weight ----- False\n",
"Parameter: 114bert.encoder.layer.6.output.dense.bias ----- False\n",
"Parameter: 115bert.encoder.layer.6.output.LayerNorm.weight ----- False\n",
"Parameter: 116bert.encoder.layer.6.output.LayerNorm.bias ----- False\n",
"Parameter: 117bert.encoder.layer.7.attention.self.query.weight ----- False\
n",
"Parameter: 118bert.encoder.layer.7.attention.self.query.bias ----- False\n",
"Parameter: 119bert.encoder.layer.7.attention.self.key.weight ----- False\n",
"Parameter: 120bert.encoder.layer.7.attention.self.key.bias ----- False\n",
"Parameter: 121bert.encoder.layer.7.attention.self.value.weight ----- False\
n",
"Parameter: 122bert.encoder.layer.7.attention.self.value.bias ----- False\n",
"Parameter: 123bert.encoder.layer.7.attention.output.dense.weight -----
False\n",
"Parameter: 124bert.encoder.layer.7.attention.output.dense.bias ----- False\
n",
"Parameter: 125bert.encoder.layer.7.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 126bert.encoder.layer.7.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 127bert.encoder.layer.7.intermediate.dense.weight ----- False\n",
"Parameter: 128bert.encoder.layer.7.intermediate.dense.bias ----- False\n",
 "Parameter: 129bert.encoder.layer.7.output.dense.weight ----- False\n",
"Parameter: 130bert.encoder.layer.7.output.dense.bias ----- False\n",
"Parameter: 131bert.encoder.layer.7.output.LayerNorm.weight ----- False\n",
"Parameter: 132bert.encoder.layer.7.output.LayerNorm.bias ----- False\n",
"Parameter: 133bert.encoder.layer.8.attention.self.query.weight ----- False\
n",
"Parameter: 134bert.encoder.layer.8.attention.self.query.bias ----- False\n",
"Parameter: 135bert.encoder.layer.8.attention.self.key.weight ----- False\n",
"Parameter: 136bert.encoder.layer.8.attention.self.key.bias ----- False\n",
"Parameter: 137bert.encoder.layer.8.attention.self.value.weight ----- False\
n",
"Parameter: 138bert.encoder.layer.8.attention.self.value.bias ----- False\n",
"Parameter: 139bert.encoder.layer.8.attention.output.dense.weight -----
False\n",
"Parameter: 140bert.encoder.layer.8.attention.output.dense.bias ----- False\
n",
"Parameter: 141bert.encoder.layer.8.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 142bert.encoder.layer.8.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 143bert.encoder.layer.8.intermediate.dense.weight ----- False\n",
"Parameter: 144bert.encoder.layer.8.intermediate.dense.bias ----- False\n",
"Parameter: 145bert.encoder.layer.8.output.dense.weight ----- False\n",
"Parameter: 146bert.encoder.layer.8.output.dense.bias ----- False\n",
"Parameter: 147bert.encoder.layer.8.output.LayerNorm.weight ----- False\n",
"Parameter: 148bert.encoder.layer.8.output.LayerNorm.bias ----- False\n",
"Parameter: 149bert.encoder.layer.9.attention.self.query.weight ----- False\
n",
"Parameter: 150bert.encoder.layer.9.attention.self.query.bias ----- False\n",
"Parameter: 151bert.encoder.layer.9.attention.self.key.weight ----- False\n",
"Parameter: 152bert.encoder.layer.9.attention.self.key.bias ----- False\n",
"Parameter: 153bert.encoder.layer.9.attention.self.value.weight ----- False\
n",
"Parameter: 154bert.encoder.layer.9.attention.self.value.bias ----- False\n",
"Parameter: 155bert.encoder.layer.9.attention.output.dense.weight -----
False\n",
"Parameter: 156bert.encoder.layer.9.attention.output.dense.bias ----- False\
n",
"Parameter: 157bert.encoder.layer.9.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 158bert.encoder.layer.9.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 159bert.encoder.layer.9.intermediate.dense.weight ----- False\n",
"Parameter: 160bert.encoder.layer.9.intermediate.dense.bias ----- False\n",
"Parameter: 161bert.encoder.layer.9.output.dense.weight ----- False\n",
"Parameter: 162bert.encoder.layer.9.output.dense.bias ----- False\n",
"Parameter: 163bert.encoder.layer.9.output.LayerNorm.weight ----- False\n",
"Parameter: 164bert.encoder.layer.9.output.LayerNorm.bias ----- False\n",
"Parameter: 165bert.encoder.layer.10.attention.self.query.weight ----- False\
n",
"Parameter: 166bert.encoder.layer.10.attention.self.query.bias ----- False\
n",
"Parameter: 167bert.encoder.layer.10.attention.self.key.weight ----- False\
n",
"Parameter: 168bert.encoder.layer.10.attention.self.key.bias ----- False\n",
"Parameter: 169bert.encoder.layer.10.attention.self.value.weight ----- False\
n",
"Parameter: 170bert.encoder.layer.10.attention.self.value.bias ----- False\
n",
 "Parameter: 171bert.encoder.layer.10.attention.output.dense.weight -----
False\n",
"Parameter: 172bert.encoder.layer.10.attention.output.dense.bias ----- False\
n",
"Parameter: 173bert.encoder.layer.10.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 174bert.encoder.layer.10.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 175bert.encoder.layer.10.intermediate.dense.weight ----- False\
n",
"Parameter: 176bert.encoder.layer.10.intermediate.dense.bias ----- False\n",
"Parameter: 177bert.encoder.layer.10.output.dense.weight ----- False\n",
"Parameter: 178bert.encoder.layer.10.output.dense.bias ----- False\n",
"Parameter: 179bert.encoder.layer.10.output.LayerNorm.weight ----- False\n",
"Parameter: 180bert.encoder.layer.10.output.LayerNorm.bias ----- False\n",
"Parameter: 181bert.encoder.layer.11.attention.self.query.weight ----- False\
n",
"Parameter: 182bert.encoder.layer.11.attention.self.query.bias ----- False\
n",
"Parameter: 183bert.encoder.layer.11.attention.self.key.weight ----- False\
n",
"Parameter: 184bert.encoder.layer.11.attention.self.key.bias ----- False\n",
"Parameter: 185bert.encoder.layer.11.attention.self.value.weight ----- False\
n",
"Parameter: 186bert.encoder.layer.11.attention.self.value.bias ----- False\
n",
"Parameter: 187bert.encoder.layer.11.attention.output.dense.weight -----
False\n",
"Parameter: 188bert.encoder.layer.11.attention.output.dense.bias ----- False\
n",
"Parameter: 189bert.encoder.layer.11.attention.output.LayerNorm.weight -----
False\n",
"Parameter: 190bert.encoder.layer.11.attention.output.LayerNorm.bias -----
False\n",
"Parameter: 191bert.encoder.layer.11.intermediate.dense.weight ----- False\
n",
"Parameter: 192bert.encoder.layer.11.intermediate.dense.bias ----- False\n",
"Parameter: 193bert.encoder.layer.11.output.dense.weight ----- False\n",
"Parameter: 194bert.encoder.layer.11.output.dense.bias ----- False\n",
"Parameter: 195bert.encoder.layer.11.output.LayerNorm.weight ----- False\n",
"Parameter: 196bert.encoder.layer.11.output.LayerNorm.bias ----- False\n",
"Parameter: 197bert.pooler.dense.weight ----- False\n",
"Parameter: 198bert.pooler.dense.bias ----- False\n",
"Parameter: 199classifier.weight ----- True\n",
"Parameter: 200classifier.bias ----- True\n"
]
}
],
"source": [
"# We can check whether the model was correctly updated\n",
"for index, (name, param) in enumerate(model.named_parameters()):\n",
" print(f\"Parameter: {index}{name} ----- {param.requires_grad}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
 "base_uri": "https://localhost:8080/",
"height": 285
},
"executionInfo": {
"elapsed": 24489,
"status": "ok",
"timestamp": 1719387042249,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "QyleqOHICBjj",
"outputId": "f2dc589c-1b8f-40e7-ef93-3ba0f3d3a3a3"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertForSequenceClassification were not initialized from the
model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias',
'classifier.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use
it for predictions and inference.\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='534' max='534' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [534/534 00:21, Epoch 1/1]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>500</td>\n",
" <td>0.474600</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
 },
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='67' max='67' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [67/67 00:01]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.4092540740966797,\n",
" 'eval_f1': 0.8141086749285034,\n",
" 'eval_runtime': 2.7437,\n",
" 'eval_samples_per_second': 388.523,\n",
" 'eval_steps_per_second': 24.419,\n",
" 'epoch': 1.0}"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Load model\n",
"model_id = \"bert-base-cased\"\n",
"model = AutoModelForSequenceClassification.from_pretrained(model_id,
num_labels=2)\n",
"tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
"\n",
"# Encoder block 10 starts at index 165 and\n",
"# we freeze everything before that block\n",
"for index, (name, param) in enumerate(model.named_parameters()):\n",
" if index < 165:\n",
" param.requires_grad = False\n",
"\n",
"# Trainer which executes the training process\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_train,\n",
" eval_dataset=tokenized_test,\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")\n",
 "trainer.train()\n",
"trainer.evaluate()"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "HJRMWsLdA913"
},
"source": [
"### [BONUS] Freeze blocks"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ADvKJjNaFAot"
},
"outputs": [],
"source": [
"# scores = []\n",
"# for index in range(12):\n",
"# # Re-load model\n",
"# model = AutoModelForSequenceClassification.from_pretrained(\"bert-base-
cased\", num_labels=2)\n",
"# tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
"\n",
"# # Freeze encoder blocks 0-index\n",
"# for name, param in model.named_parameters():\n",
"# if \"layer\" in name:\n",
"# layer_nr = int(name.split(\"layer\")[1].split(\".\")[1])\n",
"# if layer_nr <= index:\n",
"# param.requires_grad = False\n",
"# else:\n",
"# param.requires_grad = True\n",
"\n",
"# # Train\n",
"# trainer = Trainer(\n",
"# model=model,\n",
"# args=training_args,\n",
"# train_dataset=tokenized_train,\n",
"# eval_dataset=tokenized_test,\n",
"# tokenizer=tokenizer,\n",
"# data_collator=data_collator,\n",
"# compute_metrics=compute_metrics,\n",
"# )\n",
"# trainer.train()\n",
"\n",
"# # Evaluate\n",
"# score = trainer.evaluate()[\"eval_f1\"]\n",
"# scores.append(score)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
 "base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 212,
"status": "ok",
"timestamp": 1712321357732,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "dWYlHFNdLQtk",
"outputId": "a28d645b-7633-4bdf-cc51-b7a937d36508"
},
"outputs": [
{
"data": {
"text/plain": [
"[0.8541862652869239,\n",
" 0.8525519848771267,\n",
" 0.8514664143803217,\n",
" 0.8506616257088847,\n",
" 0.8398104265402844,\n",
" 0.8391345249294448,\n",
" 0.8377358490566037,\n",
" 0.8433962264150944,\n",
" 0.8258801141769743,\n",
" 0.816247582205029,\n",
" 0.7917485265225934,\n",
" 0.7019400352733686]"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# scores"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 410
},
"executionInfo": {
"elapsed": 1383,
"status": "ok",
"timestamp": 1712388601684,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
 },
"id": "bf3PIvKhOBJ-",
"outputId": "aadf0ff7-e4ab-4d62-87b5-cbf1ecdea8da"
},
"outputs": [
{
"data": {
"image/png":
"iVBORw0KGgoAAAANSUhEUgAAArwAAAGJCAYAAABo5eDAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIH
ZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/
bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACUfklEQVR4nOzdd1iT19sH8G8SSMJGmU5w4hYFQXGvupU66q
iK+2fVaktrK3VVW7XT2m2tqK21Ql2tEwdK3WJxD3CLWgUXWxJIzvuHJa8RkCEkIXw/15VLc/
KM+86TcfPkPOdIhBACRERERERmSmrsAIiIiIiIShMLXiIiIiIyayx4iYiIiMisseAlIiIiIrPGgpeIiIiIz
BoLXiIiIiIyayx4iYiIiMisseAlIiIiIrPGgpeIiIiIzBoLXjKKtLQ0jBs3Du7u7pBIJHjrrbcAAAkJCRg4
cCCcnJwgkUiwZMkSo8ZZFPnlRGVTVFQUJBIJoqKijB1KLp6enhg1alSpbPvGjRuQSCT44osvSmX75mTUqFH
w9PQs1roffvghJBJJyQZkgvi5SKaCBS+VmFWrVkEikeR7O3r0qG7ZhQsXYtWqVXjjjTewevVqjBgxAgDw9t
tvY+fOnQgJCcHq1avRvXv3Eo9z4cKF+PPPP0tlu3nllBdPT898n6fMzMwSj82YcgrH/
G5hYWHGDtHo8nqOKlasiJYtW2LNmjXGDq/
MedHr7dmbKf4xYwijRo3Sex7s7e3RtGlTfPnll1CpVCW6r6J8LhKVJgtjB0DmZ/
78+ahRo0au9tq1a+v+v3fvXrRs2RJz587VW2bv3r3o168f3n333VKLb+HChRg4cCACAwNLdLv55ZQfb29vv
PPOO7na5XJ5icZlKqZOnYoWLVrkam/
VqpURojFNzz5HDx8+RHh4OIYPH46kpCRMnjzZyNGVHatXr9a7/+uvv2L37t252uvXr/9S+/
n555+h1WqLte6sWbMwY8aMl9r/
y1AoFFi+fDkAICkpCRs2bMC7776L48ePl+gfoUX9XCQqLSx4qcT16NEDvr6+L1wmMTERDRo0yLPd0dGxlCI
rXfnllJ8qVapg+PDhhV4+IyMD1tbWxQnNJLRt2xYDBw40dhhGk56eDhsbmxcu8/xz9MYbb6BmzZr4/
fffWfAWwfPvq6NHj2L37t0Fvt+K+h6ztLQsVnwAYGFhAQsL430FW1hY6D0fkyZNgr+/
P8LDw7F48WJUrly52NvWarVQq9VQKpVF/
lwsSHZ2NrRardmeGKDSwy4NZFA5P91ev34d27Zt0/2kltMdQgiB77//
XteeIykpCW+99RaqVasGhUKB2rVr49NPP811dkWr1eLrr79G48aNoVQq4eLigu7du+Off/
4B8PSnzvT0dPzyyy+6fRTUFzIxMRFjx46Fm5sblEolmjZtil9+
+aXAnG7cuFHs56lDhw5o1KgRYmJi0K5dO1hbW+ODDz4oVDw56+f3M+6qVauK9Lw+26dz2bJlqFWrFhQKBVq
0aIHjx48XO8e8SCQSTJkyBX/+
+ScaNWoEhUKBhg0bIiIiIteyd+7cwdixY1G5cmUoFArUqFEDb7zxBtRqtW6Za9euYdCgQahYsSKsra3RsmV
LbNu2Lde2bt++jcDAQNjY2MDV1RVvv/12vj/
tHjt2DN27d4eDgwOsra3Rvn17HDp0SG+ZnP6ZFy5cwLBhw1ChQgW0adOmyM+HXC5HhQoVClUYFTbXzMxMfP
jhh6hbty6USiUqVaqE/v374+rVq/
luWwiBCRMmQC6XY+PGjQCArKwszJs3D3Xq1IFSqYSTkxPatGmD3bt3l0isOe+rP/
74AwsWLEDVqlWhVCrRuXNnXLlypcB9FORF77G//voLvXr10r22atWqhY8+
+ggajUZvG8/34S3KeyWvPrxFef1HRUXB19cXSqUStWrVwk8//fRS/
YKlUik6dOigywMAVCoV5s6di9q1a0OhUKBatWp47733cr03cuJes2YNGjZsCIVCgYiIiBd+Lhbmc+zZ53PJ
kiW65/PChQu6XC9duoThw4fDwcEBLi4umD17NoQQuHXrFvr16wd7e3u4u7vjyy+/
1Nu2Wq3GnDlz4OPjAwcHB9jY2KBt27bYt29fvjEU5vMvNjYWr732GlxcXGBlZQUvLy/
MnDlTb5k7d+5gzJgxcHNz0x3jFStWFPWQURHxDC+VuOTkZDx48ECvTSKRwMnJCfXr18fq1avx9ttvo2rVqr
qf9Js1a6br39W1a1eMHDlSt25GRgbat2+PO3fu4H//
+x+qV6+Ow4cPIyQkBHfv3tW7sG3s2LFYtWoVevTogXHjxiE7OxsHDhzA0aNH4evri9WrV2PcuHHw8/
PDhAkTAAC1atXKN5cnT56gQ4cOuHLlCqZMmYIaNWpg3bp1GDVqFJKSkjBt2rR8c3JxcXnh85SVlZXrebK2t
tadYXr48CF69OiBIUOGYPjw4XBzcytUPAAwc+ZMjBs3Tm/bv/32G3bu3AlXV9ciP68A8PvvvyM1NRX/
+9//IJFI8Nlnn6F///
64du1aoc50paam5soXgO4CxRwHDx7Exo0bMWnSJNjZ2eGbb77BgAEDEB8fDycnJwDAv//+Cz8/
PyQlJWHChAmoV68e7ty5g/Xr1yMjIwNyuRwJCQkICAhARkYGpk6dCicnJ/zyyy/
o27cv1q9fj1dffRXA02PcuXNnxMfHY+rUqahcuTJWr16NvXv35op179696NGjB3x8fDB37lxIpVKsXLkSnT
p1woEDB+Dn56e3/KBBg1CnTh0sXLgQQogiPUePHj3C77//
jnPnziE0NPSF6xU2V41Gg969eyMyMhJDhgzBtGnTkJqait27d+PcuXN5vhc0Gg3GjBmD8PBwbNq0Cb169QL
wtGBbtGiR7v2UkpKCf/75BydOnEDXrl1fOtYcn3zyCaRSKd59910kJyfjs88+w+uvv45jx44V+HwWJK/
3GPD0egRbW1sEBwfD1tYWe/fuxZw5c5CSkoLPP/+8wO2+zHulMK//
kydPonv37qhUqRLmzZsHjUaD+fPnF/
iZU5CcP3qcnJyg1WrRt29fHDx4EBMmTED9+vVx9uxZfPXVV7h06VKu6yD27t2LP/
74A1OmTIGzszMqVaqU7+diYT/HcqxcuRKZmZmYMGECFAoFKlasqHts8ODBqF+/
Pj755BNs27YNH3/8MSpWrIiffvoJnTp1wqeffoo1a9bg3XffRYsWLdCuXTsAQEpKCpYvX46hQ4di/
PjxSE1NRWhoKLp164bo6Gh4e3vrxVCYY3rmzBm0bdsWlpaWmDBhAjw9PXH16lVs2bIFCxYsAPD09d+yZUvd
HwkuLi7YsWMHxo4di5SUFF7UV5oEUQlZuXKlAJDnTaFQ6C3r4eEhevXqlWsbAMTkyZP12j766CNhY2MjLl2
6pNc+Y8YMIZPJRHx8vBBCiL179woAYurUqbm2q9Vqdf+3sbERQUFBhcppyZIlAoD47bffdG1qtVq0atVK2N
raipSUlAJzyouHh0eez9PcuXOFEEK0b99eABBLly4tdjzPOnTokLC0tBRjxozRtRX2eb1+/
boAIJycnMSjR490y/31118CgNiyZcsLc923b1+
+rwsA4u7du7plAQi5XC6uXLmiazt9+rQAIL799ltd28iRI4VUKhXHjx/Ptb+cY/
3WW28JAOLAgQO6x1JTU0WNGjWEp6en0Gg0Qoj/
f07/+OMP3XLp6emidu3aAoDYt2+fbrt16tQR3bp103s9ZWRkiBo1aoiuXbvq2ubOnSsAiKFDh77wuSnoOZJ
KpWLBggW5lvfw8NB7DRc21xUrVggAYvHixfk+bznH+/
PPPxdZWVli8ODBwsrKSuzcuVNv+aZNmxb69f6swsaa85zUr19fqFQq3bJff/
21ACDOnj1b6H1OnjxZPP91l997TIinx/R5//vf/4S1tbXIzMzUtQUFBQkPDw/d/aK8V3JeI88q7Ou/
T58+wtraWty5c0fXdvnyZWFhYZFrm3kJCgoSNjY24v79++L+/
fviypUrYuHChUIikYgmTZoIIYRYvXq1kEqlesdJCCGWLl0qAIhDhw7pxS2VSsX58+dz7Suvz8XCfo7lPJ/
29vYiMTFRbxs5z9+ECRN0bdnZ2aJq1apCIpGITz75RNf+
+PFjYWVlpfeeyc7O1ntd5Szn5uam9zlZlGParl07YWdnJ27evKm33Wc/
L8aOHSsqVaokHjx4oLfMkCFDhIODQ56vPSoZ7NJAJe7777/
H7t279W47duwo9vbWrVuHtm3bokKFCnjw4IHu1qVLF2g0Guzfvx8AsGHDBkgkkjwvjijuz3zbt2+Hu7s7hg
4dqmuztLTE1KlTkZaWhr///rt4SQHw9/
fP9Tw9e2ZboVBg9OjRLx3PvXv3MHDgQHh7e+OHH37QtRf2ec0xePBgVKhQQXe/
bdu2AJ7+PF0Yc+bMyZXv7t279c7WAECXLl30zjQ2adIE9vb2uv1otVr8+eef6NOnT559xXOO9fbt2+Hn56f
XlcDW1hYTJkzAjRs3cOHCBd1ylSpV0us7a21trfsFIMepU6dw+fJlDBs2DA8fPtQ9X+np6ejcuTP279+fq4
vNxIkTC/Xc5PUchYeHY+jQoZg5cya+/
vrrF65X2Fw3bNgAZ2dnvPnmm7m28fx7RK1WY9CgQdi6dSu2b9+OV155Re9xR0dHnD9/
HpcvXy5SjoWNNcfo0aP1+msW9XX3Inm9xwDAyspK9/+cs+5t27ZFRkYGYmNjC9zuy7xXCnr9azQa7NmzB4G
BgXr9bGvXro0ePXoUuP0c6enpcHFxgYuLC2rXro0PPvgArVq1wqZNmwA8/XyoX78+6tWrp/
f50KlTJwDI9dN/+/btC91Xt6ifYwMGDMj37PWzv2TJZDL4+vpCCIGxY8fq2h0dHeHl5aX3/
MtkMt3rSqvV4tGjR8jOzoavry9OnDiRaz8FHdP79+9j//
79GDNmDKpXr663bs57SwiBDRs2oE+fPhBC6D2v3bp1Q3Jycp77ppLBLg1U4vz8/Aq8aK0oLl+
+jDNnzuT7gZeYmAjg6c9xlStXzlVAvYybN2+iTp06kEr1/zbMubr75s2bxd62s7MzunTpku/
jVapUyXVhRlHjyc7OxmuvvQaNRoONGzdCoVDoHivs85rj+Q/xnA//
x48f55vDsxo3bvzCfPPbT86+cvZz//59pKSkoFGjRi/czs2bN+Hv75+r/
dnnqlGjRrh58yZq166dq+Dz8vLSu59T2AUFBeW7z+TkZL0vxbxGK3mR55+j1157DcnJyZgxYwaGDRuW77Eq
bK5Xr16Fl5dXofoEL1q0CGlpadixY4eub+ez5s+fj379+qFu3bpo1KgRunfvjhEjRqBJkyYv3G5hY83xsq+
7F8nrPQYA58+fx6xZs7B3716kpKToPZacnFzgdl8m5oJe/
4mJiXjy5IneqDc58mrLj1KpxJYtWwBA1we+atWquscvX76MixcvFvrzoSiv9aJ+jr1o288/
Xw4ODlAqlXB2ds7V/vDhQ722X375BV9+
+SViY2ORlZX1wv0VdExzCt8XfS7dv38fSUlJWLZsGZYtW5bnMs8/r1RyWPCSydNqtejatSvee+
+9PB+vW7eugSMyjGfPMhXX9OnTceTIEezZs0fvywwo+vMqk8nyXE4Uom9qURhqP0WVc/b2888/z9W/
L4etra3e/ZI4hp07d8bWrVsRHR2t6z9rCN26dUNERAQ+++wzdOjQAUqlUu/xdu3a4erVq/jrr7+wa9cuLF+
+HF999RWWLl2aq//
4yyjN10NexycpKQnt27eHvb095s+fj1q1akGpVOLEiRN4//33CzUM2cvEbMj32Yv+ANVqtWjcuDEWL16c5+
PVqlXTu18Sr/X8vGjbeT1fhXkOf/
vtN4waNQqBgYGYPn06XF1dIZPJsGjRojwv4CyJ45Lz2hk+fHi+fzgX9AcjFR8LXjJ5tWrVQlpaWoFnB2vVq
oWdO3fi0aNHLzzLW5TuDR4eHjhz5gy0Wq3e2YicnzU9PDwKva2SUJR4wsLCsGTJEixZsgTt27fPta3CPq+m
xsXFBfb29jh37twLl/
Pw8EBcXFyu9uefKw8PD5w7dw5CCL3XxvPr5vzMbG9vb9DnLDs7G8DTGavyU9hca9WqhWPHjiErK6vAi6dat
myJiRMnonfv3hg0aBA2bdqU68xwxYoVMXr0aIwePRppaWlo164dPvzwwxcWvIWN1ViioqLw8OFDbNy4UXeB
EwBcv37diFH9P1dXVyiVyjxHqiiJ0Sty1KpVC6dPn0bnzp1LfEY4U/
hcXb9+PWrWrImNGzfq5Vfc8YJr1qwJAC/
8XHJxcYGdnR00Gk2Z+9w1B+zDSybvtddew5EjR7Bz585cjyUlJekKggEDBkAIgXnz5uVa7tm/
wm1sbJCUlFSofffs2RP37t1DeHi4ri07OxvffvstbG1t8ywkS1Nh4zl37hzGjRuH4cOH57riOUdhn1dTI5V
KERgYiC1btuiGm3tWzrHu2bMnoqOjceTIEd1j6enpWLZsGTw9PXX9DXv27Il///
0X69ev1y2XkZGR6ydHHx8f1KpVC1988UWexef9+/dLJL/
nbd26FQDQtGnTfJcpbK4DBgzAgwcP8N133+XaRl5nqrp06YKwsDBERERgxIgRemc3n/
952NbWFrVr1y5wpq7CxmosOWfynn0+1Gq1Xv93Y8o5M/vnn3/
i33//1bVfuXLlpa6VeN5rr72GO3fu4Oeff8712JMnT5Cenl7sbZvC52pex/
nYsWN6r8uicHFxQbt27bBixQrEx8frPZazD5lMhgEDBmDDhg15Fsal9RlCT/
EML5W4HTt25HlhR0BAgO6v4KKYPn06Nm/ejN69e2PUqFHw8fFBeno6zp49i/
Xr1+PGjRtwdnZGx44dMWLECHzzzTe4fPkyunfvDq1WiwMHDqBjx46YMmUKgKeFy549e3SDq9eoUSPPPoUAM
GHCBPz0008YNWoUYmJi4OnpifXr1+PQoUNYsmQJ7OzsipzPyyhsPDkX4rRr1w6//
fab3jZyjkNhn9eScuDAgTynTW7SpEmRf8ZbuHAhdu3ahfbt2+uGTLp79y7WrVuHgwcPwtHRETNmzMDatWvR
o0cPTJ06FRUrVsQvv/
yC69evY8OGDbozS+PHj8d3332HkSNHIiYmRjec0vMTEEilUixfvhw9evRAw4YNMXr0aFSpUgV37tzBvn37Y
G9vr+sTWVzPPkePHj3C5s2b8ffff2PIkCGoV69evusVNteRI0fi119/
RXBwMKKjo9G2bVukp6djz549mDRpEvr165dr24GBgVi5ciVGjhwJe3t7/
PTTTwCABg0aoEOHDvDx8UHFihXxzz//
YP369br32cvGaiwBAQGoUKECgoKCMHXqVEgkEqxevdroXWqe9eGHH2LXrl1o3bo13njjDWg0Gnz33Xdo1Kg
RTp06VSL7GDFiBP744w9MnDgR+/btQ+vWraHRaBAbG4s//
vgDO3fuLPa1Gqbwudq7d29s3LgRr776Knr16oXr169j6dKlaNCgwQt/
TXmRb775Bm3atEHz5s0xYcIE1KhRAzdu3MC2bdt0x+WTTz7Bvn374O/vj/
Hjx6NBgwZ49OgRTpw4gT179uDRo0clmCXpMfCoEGTGXjQsGQCxcuVK3bJFGZZMiKfDFoWEhIjatWsLuVwun
J2dRUBAgPjiiy+EWq3WLZednS0+//
xzUa9ePSGXy4WLi4vo0aOHiImJ0S0TGxsr2rVrJ6ysrASAAocoS0hIEKNHjxbOzs5CLpeLxo0b6+VSUE55K
WjZ9u3bi4YNGxY7nvyGPXv+OBTmeX12mKrn4Zmh1PJT0LBkz66f3/F/
fhguIYS4efOmGDlypHBxcREKhULUrFlTTJ48WW+ooatXr4qBAwcKR0dHoVQqhZ+fn9i6dWuu7d+8eVP07dt
XWFtbC2dnZzFt2jQRERGhNyxZjpMnT4r+/fsLJycnoVAohIeHh3jttddEZGSkbpmcIZPu37//
wufmRc+RXC4X9erVEwsWLNB7jef3fBQ214yMDDFz5kxRo0YNYWlpKdzd3cXAgQPF1atXhRD5H+8ffvhBABD
vvvuuEEKIjz/+WPj5+QlHR0dhZWWVb6x5KUysOc/JunXr9Npz4svrPZif/
IYly+89dujQIdGyZUthZWUlKleuLN577z2xc+fOXK+H/
IYlK8x7Jb9hyQr7+o+MjBTNmjUTcrlc1KpVSyxfvly88847QqlU5vMs/
L+cYckKolarxaeffioaNmwoFAqFqFChgvDx8RHz5s0TycnJBcadE3ten3WF+Rx70fOZ33ssv9yeP95arVYs
XLhQeHh4CIVCIZo1aya2bt36UsdUCCHOnTsnXn31Vd1r28vLS8yePTtX7pMnTxbVqlXTvQc7d+4sli1blms
fVHIkQpjQn61ERERULIGBgcUaKo6oPGAfXiIiojLmyZMnevcvX76M7du35zmEHBEBPMNLRERUxlSqVAmjRo
1CzZo1cfPmTfz4449QqVQ4efIk6tSpY+zwiEwOL1ojIiIqY7p37461a9fi3r17UCgUaNWqFRYuXMhilygfP
MNLRERERGaNfXiJiIiIyKyx4CUiIiIis2b0Przff/89Pv/8c9y7dw9NmzbFt99+Cz8/
v3yXX7JkCX788UfEx8fD2dkZAwcOxKJFi/Tmeb9z5w7ef/997NixAxkZGahduzZWrlxZ6EGytVot/
v33X9jZ2ZX4lIpERERE9PKEEEhNTUXlypULnrTGiGMAi7CwMCGXy8WKFSvE+fPnxfjx44Wjo6NISEjIc/
k1a9YIhUIh1qxZI65fvy527twpKlWqJN5+
+23dMo8ePRIeHh5i1KhR4tixY+LatWti586d4sqVK4WO69atWy8cKJ833njjjTfeeOONN9O43bp1q8Dazqg
Xrfn7+6NFixa6ed21Wi2qVauGN998EzNmzMi1/
JQpU3Dx4kVERkbq2t555x0cO3YMBw8eBPB02spDhw7hwIEDxY4rOTkZjo6OuHXrFuzt7Yu9ncLSCoGk5FQ4
OthBagZnlM0pH+ZimswpF8C88mEupom5mC5zysfQuaSkpKBatWpISkqCg4PDC5c1WpcGtVqNmJgYhISE6Nq
kUim6dOmCI0eO5LlOQEAAfvvtN0RHR8PPzw/Xrl3D9u3bMWLECN0ymzdvRrdu3TBo0CD8/
fffqFKlCiZNmoTx48fnG4tKpYJKpdLdT01NBQDY2tnB1gBzegshoBES2NnZmkUXCnPKh7mYHpGpRsobP8JO
q4Xt0kmQWimMHdJLM5djAzAXU8VcTJc55WPoXLT/
nbMtzL6MVvA+ePAAGo0Gbm5ueu1ubm6IjY3Nc51hw4bhwYMHaNOmDYQQyM7OxsSJE/HBBx/olrl27Rp+/
PFHBAcH44MPPsDx48cxdepUyOVyBAUF5bndRYsWYd68ebnak5JToRGGe/
ElpaQZbF+GYE75MBcTkqkGbt4HACSnpAFqtZEDKjll/
tg8g7mYJuZiuswpH0PlkpqSWuhljX7RWlFERUVh4cKF+OGHH+Dv748rV65g2rRp+OijjzB79mwAT7tF+Pr6
YuHChQCAZs2a4dy5c1i6dGm+BW9ISAiCg4N193NOkTs62MHe3jBneJNS0uBoX/b/ugPMKx/mYnqEXI2k//
7vYGcLqbV5nOE1h2MDMBdTxVxMlznlY+hcZJLC98o1WsHr7OwMmUyGhIQEvfaEhAS4u7vnuc7s2bMxYsQIj
Bs3DgDQuHFjpKenY8KECZg5cyakUikqVaqEBg0a6K1Xv359bNiwId9YFAoFFIrcX5pSicQgfVC0//0rMdD+
Sps55cNcTI94JvaynksOczk2AHMxVczFdJlTPobOpSj7MNo4vHK5HD4+PnoXoGm1WkRGRqJVq1Z5rpORkZF
r2AmZTAbg6V8VANC6dWvExcXpLXPp0iV4eHiUZPhEREREVEYYtUtDcHA
wgoKC4OvrCz8/
PyxZsgTp6ekYPXo0AGDkyJGoUqUKFi1aBADo06cPFi9ejGbNmum6NMyePRt9+vTRFb5vv/
02AgICsHDhQrz22muIjo7GsmXLsGzZMqPlSURERETGY9SCd/Dgwbh//z7mzJmDe/
fuwdvbGxEREboL2eLj4/
XO6M6aNQsSiQSzZs3CnTt34OLigj59+mDBggW6ZVq0aIFNmzYhJCQE8+fPR40aNbBkyRK8/
vrrBs+PiIiIiIzPqOPwmqqUlBQ4ODggOTnZYOPwPk5ORQUzGIMPMK98mIvpEZlqJI36GlqtFo6/
vAWZGQxLZi7HBmAupoq5mC5zysfQuRSlXitTozQQEUmUcjisfRePk1MhUcqNHQ4REZUBRrtojYiIiIjIEHi
Gl4iIiIiKRKPRIDU1FcnJybrb46QkJN5/
gDGjRxk7vFxY8BJRmSJUWUiZ9jOg0UB8+z+A3RqIiIpEo9EgJSVFr1gt6i01Ne9ZziQSCUYFjYT0v9GzTAU
LXiIqW7QCmrg7uv8TEZUn2dnZL12spqWV3NS/
CoUCDg4Oupu1jQ1UKhUsrK1LbB8lgQUvERERUSkRQiAzMxMZGRl53tLS03HvXiKys1SFKmRLslhVKpV6xWp
xbs/
OVJszSoOVlVWJxVhSWPASERFRuSOEQFZWFjIyMvDkyZN8C9KSuJWGgopVR0fHAotVubz8dAljwUtERERGJ4
SASqVCZmYmVCqV3v+fbct48gSPHidBAoHMlyxUNRqNQXOUy+WwtrbWu1lZWcHK2hpOTk5wLMKZ1fJUrJYEF
rxERETllBAC2dnZBRaZhmhTq9VGex6kUilsbGxyFaLPF6cvc7OysoKFRe6yy5wmnjBlLHiJiIjMlBAChw4d
wvLQUBw6dAhZanWuwtNUJ1xVKBRQKpVQKBR6/1cqlZDJLGBnbwebEipGLS0tIWGxadZY8BJRmSNxsDbZL2k
iU5CYmIhff/
0Vy5cvR1xcXKHXs7CwyLfINGSbXC7PtwDlGVEqDha8RFSmSKzkcNz0wdOpha3Yh40oh0ajwa5du7B8+XJs3
rwZ2dnZAABra2sMHjwY3Xv2RvVqVWClVOZZZCoUCshMbOxUopLCgpeIiKgMu3HjBlasWIGVK1fi9u3bunZ/
f3+MHTsWgwcPhq2dHc+KUrnGgpeIiKiMUalU+PPPPxEaGoo9e/
bouvhUrFgRI0aMwNixY9G4cWPd8lp2AaJyjgUvEZUpQpWF1PdXAdkaiC/
GcGphKlfOnTuH0NBQrF69Gg8fPtS1d+nSBePGjUO/fv2gVCqNGCGRaWLBS0Rli1Yg+/
QN3f+JzF1qairCw8OxfPlyHDt2TNdepUoVjBkzBqNHj0aNGjWMGCGR6WPBS0REZGKEEDh69CiWL1+O8PBwp
KenA3g6ikLfvn0xduxYdOvWjReZERUSC14iIiITcf/
+faxevRqhoaG4cOGCrr1u3boYN24cRo4cCTc3NyNGSFQ2seAlIiIyIq1Wiz179mD58uX4888/
kZWVBQCwsrLCa6+9hnHjxqF169acGIHoJbDgJSIiMoL4+HisXLkSK1asQHx8vK7d19cX48aNw5AhQ+Dg4GD
ECInMBwteIiIiA1Gr1di8eTOWL1+OXbt26YYTc3R0xPDhwzF27Fh4e3sbN0giM8SCl4jKHqUlwAEaqAy5cO
ECQkND8euvv+LBgwe69o4dO2LcuHF49dVXYWVlZcQIicwbC14iKlMkVnLc+WwAdu3eg+bHDsPXxwf29vbGD
osol7S0NPzxxx8IDQ3F4cOHde2VKlXC6NGjMWbMGNSqVcuIERKVHyx4iajMSE9Px4cffoivvvoKGo1G1+7l
5QVfX1/
dzdvbG7a2tkaMlMorIQSOHz+O5cuXY+3atUhLSwMAyGQy9O7dG+PGjUP37t1hYcGvXyJD4juOiMqEHTt2YN
KkSbhx4wYAwLdFC9xPTMTNmzcRFxeHuLg4rFmzBgAglUpRv359vSK4adOm/MmYSs3Dhw/x22+/
Yfny5Th37pyuvXbt2hg7diyCgoJQqVIlI0ZIVL6x4CUik3bv3j289dZbCA8PBwBUq1YNn/
sNQBfPRnBcMAKPUpIx7+gpPLkUi/sH9+HE8WjcuXMH58+fx/nz5/
HLL78AeHqGrVGjRnpFcOPGjaFQKIyZHpVhWq0We/
fuRWhoKDZu3Ai1Wg0AUCqVGDhwIMaNG4d27dpxODEiEyA1dgAA8P3338PT0xNKpRL+/
v6Ijo5+4fJLliyBl5cXrKysUK1aNbz99tvIzMzMc9lPPvkEEokEb731VilETkSlRavVYtmyZahfvz7Cw8Mh
lUrx9ttv43zMKXR5aA3EXAM0Ak7OzvhBrcAKz6bYOfJt1PhtK948cgaz/9yB6fM/
Qq9eveDq6gqNRoPTp08jNDQUb7zxBlq0aAE7Ozv4+vpi4sSJWL58OU6dOqUbA5UoP7dv38bHH3+MWrVqoWv
XrggLC4NarUazZs3w/fff4+7du1i9ejXat2/
PYpfIRBj9DG94eDiCg4OxdOlS+Pv7Y8mSJejWrRvi4uLg6uqaa/nff/
8dM2bMwIoVKxAQEIBLly5h1KhRkEgkWLx4sd6yx48fx08//
YQmTZoYKh0iKgEXLlzAhAkTcOjQIQBA8+bNsWzZMvj4+EA8UePRM8tKJRIMquKMP+48gForcPBhCo5KJMgW
1pA2aIdDk6bCv6Id7ty5g3/++Ufv9vDhQ8TExCAmJgY//
fQTAEChUMDb21vvTHC9evXY57Kcy8rKwtatW7F8+XJERERAq9UCABwcHPD6669j7NixaN68uZGjJKL8GP0T
fPHixRg/fjxGjx4NAFi6dCm2bduGFStWYMaMGbmWP3z4MFq3bo1hw4YBADw9PTF06FAcO3ZMb7m0tDS8/
vrr+Pnnn/Hxxx+XfiJE9NIyMzOxYMECfPrpp8jKyoKNjQ0++ugjvPnmmy8sOOc18MQfd/5/
qKdsISAFUM1aAS87K0gkElStWhVVq1ZFYGAggKcXF928eTNXEZycnIxjx47pfaZYW1ujWbNmekVw3bp1IZW
axI9kVIouX76MDX+E4ddff0ViYqKuvX379hg7diwGDBgAa2trI0ZIRIVh1IJXrVYjJiYGISEhujapVIouXb
rgyJEjea4TEBCA3377DdHR0fDz88O1a9ewfft2jBgxQm+5yZMno1evXujSpUuBBa9KpYJKpdLdT0lJAQBoh
YBWlP5gnzkDjwshoC31vZU+c8qHuRjO3r17MemNN3D58mUAQK/evfHdd9+hevXqAKB7L4pn3pPiv/
doXTsr9HCrgF2Jj6H572GpBNjUsgEcLC3yfR9X9/BAdQ8P9B8wQLe9q1ev4p9//kHMP//
ozv6mpaXh0KFDujPOAGBrawsfH5+nt/+K4Fq1ahXrJ2xTPzZF8TK5CCGQnZ0NlUoFtVqtuz1/
v6C2rIKWe+6+Wq2GOo99ZmZm6s2A5ubmhqCgIIweMwZ169bVtRvie+Jl8TVmuswpH0PnUpT3nlEL3gcPHkC
j0cDNzU2v3c3NDbGxsXmuM2zYMDx48ABt2rTRfThOnDgRH3zwgW6ZsLAwnDhxAsePHy9UHIsWLcK8efNytS
clp0IjDNf/
KiklzWD7MgRzyoe5lJ6HDx9i7uxZCFv7OwDAzd0diz75FH369oNEIsHj5FT9FTLVuv8mp6YBWU/
vT6hcETsSHv//cgKYdOISVjWuAWd54T/
qnFzc0K1HL3Tr0QvA077EV69cwcmTJ3H61EmcOnkSZ8+eQVpaGv7++2/8/
fffunUdHBzQ1Nsb3t7N4N3s6a1ateqFLoJN7dgAT08IpKWlIT09DWmpaUhLS0NaWup//z69pT/7//
R0ZGZm6grHrCw11OosZGU9LSKz1FlQZ6l1/6pVKqizsnRFqqmRSqXo0vUVDB85El27vgJLS0sAyP26LCNM8
TVWXOaUC2Be+Rgql9SUwr8Pjd6loaiioqKwcOFC/PDDD/
D398eVK1cwbdo0fPTRR5g9ezZu3bqFadOmYffu3VAqlYXaZkhICIKDg3X3U1JSUK1aNTg62MHe3q60UtERQ
iApJQ2O9rZmcYGDOeXDXEo3nl9//RXT330XDx8+hEQiwf/+9z8sXLQIDg4O+a8nVyPpv/
872NlCav10lIU+9rZofPUuzqZkYFFDT7RxdsDAoxfQLeYyNrVsgKaOxR+X16mFD/
xa+OjuZ2dnIzY2Vu9M8KlTp5CcnIz9f/+N/c8UwU5OTk/
PAD9zJrhKlSp6x6Akj41arUZq6tOCNDU1Ndct/
fn2Z+7neiw11agX8UkkEigUCsjlcr1bXm3y59oUCgXklpYvXM8yn23ltFlaWqKCkwvq1q5pEu+Zl2Fq7/+X
YU65AOaVj6FzkUnKyBleZ2dnyGQyJCQk6LUnJCTA3d09z3Vmz56NESNGYNy4cQCAxo0bIz09HRMmTMDMmTM
RExODxMREvYsHNBoN9u/fj++++w4qlQoymUxvmwqFIs+hiaQSCaQGOGA5p/
0lBtpfaTOnfJhL6bh8+TImTpyIvXv3AgAaNWqEZcuWoVWrVgWuK56JXS8XiQTLmtdF1IMkvF+3GiQSCY53a
obAIxfQ5u/T+LWFFwZUcSmR+OWWlmjSuDGaNG6MMf9df5CVlYXz58/
r9Qc+c+YMHj58iF07d2LXzp269d3c3PSHR2vSBEnJqfhXInIVnfkVrvk9VlpnSa2srGBnZwdbW1vY2dnluu
W029raQkikcLC3g/
KZYvL5wrIwhaxMJjNqAaAVAo+TU03iPfOyTOn9/7LMKRfAvPIxdC5F2YdRC165XA4fHx9ERkbqLiTRarWIj
IzElClT8lwnIyMj14UiOQWsEAKdO3fG2bNn9R4fPXo06tWrh/fffz9XsUtEhqNWq/HZZ5/h448/
hkqlglKpxNy5c/HOO+/ofiouiMRKjgp7P35aiFjJ9R5r6WSPlk7/
P81wNWslDrRvitExlzDw6EV8WD8Ds+tXL5UPYktLS3h7e8Pb21v3B7lKpcLZs2f1iuBz584hISEB27Ztw7Z
t20o8jhxKpfKFBWphitdn7xd2lIqcIrGCg12Z//ImIvNh9C4NwcHBCAoKgq+vL/z8/
LBkyRKkp6frRm0YOXIkqlSpgkWLFgEA+vTpg8WLF6NZs2a6Lg2zZ89Gnz59IJPJYGdnh0aNGuntw8bGBk5O
TrnaichwDh06hAkTJuDChQsAgK5du+LHH39ErVq1SnW/1hYyhPnVQxMHG8w6fwNnU9Lxi68XbCxK/
49fhUKhO4ub48mTJzh9+rReEXzx4kVYWloWuRDN7zFbW9tC/
wFBRFQeGL3gHTx4MO7fv485c+bg3r178Pb2RkREhO5Ctvj4eL0zurNmzYJEIsGsWbNw584duLi4oE+fPliw
YIGxUiCiF0hKSsL777+PZcuWAQBcXFzw1VdfYdiwYQb7uVoikWBmvepoZG+N4cfj0DrqFP5q1RAeNoXr51+
SrKys0LJlS7Rs2VLXlq3RIDk1nWdFiYhKiUSIMjCeioGlpKTAwcEBycnJsLe3L3iFl2RuPwGaUz7MpfiEEP
jjjz8wbdo0XT/9MWPG4LPPPoOTk1Pxt6vOQuqCdcjKyobj3CGQKeQFr/SMs8np6Hv4HNI1Wmxo2QBtnfO/
QM5Q+DozTczFNJlTLoB55WPoXIpSr3HUdCIqcTdu3ECvXr0wZMgQJCQkwMvLC1FRUQgNDX2pYhcAoBHI2n8
eOBIH3aC7RdDYwQbHOzVHAztrdN5/Bj9fv/
ty8RARkcljwUtEJSY7OxtffPEFGjZsiB07dkAul2Pu3Lk4ffo02rdvb+zwdJwVltjdtjHG1XDHhBOX8eapK
8jW8scuIiJzZfQ+vERkHo4fP44JEybg1KlTAIB27drhp59+Qr169YwbWD4spVL80KwOGtvbYOrpq7iYkoE/
WtZHRTkv9iIiMjc8w0tELyU1NRXTpk1Dy5YtcerUKVSoUAHLly/
Hvn37TLbYfdYbtSpjd9vGOJWcBr+9J3EhJd3YIRERUQljwUtExfbXX3+hQYMG+Oabb6DVajFs2DDExsZi7N
ixucbLNmUdXBxxvFMzWMmkaLnvFLbefWjskIiIqASVnW8kIjIZd+7cQf/+/
REYGIjbt2+jZs2a2LlzJ9asWQNXV1djh1csNWyscLiDNzq7OqLv4fP4NO4WOIgNEZF5YMFLRIWm0Wjw3Xff
oX79+ti0aRMsLCwwY8YMnD17Fq+88oqxw3tpdpYW2NCyAWbVq44Z565j+PFYPNFojB0WERG9JF60RkSFcvr
0aUyYMAHR0dEAgJYtW2LZsmVo3LixYQNRWsJx2xwkpaQCypK/wEwqkWB+Q080crDBqH/
icCntNP5s1RBVrBQlvi8iIjIMnuElohfKyMjA+++/Dx8fH0RHR8Pe3h7ff/
89Dh48aPhiF09nTZNYyQGlvFRnanutqgsOdfBGQmYWfPeexLFHKaW2LyIiKl0seIkoXzt37kSjRo3w2WefQ
aPRYMCAAbh48SImTZoEmUxm7PBKXTNHWxzv1Aw1bZRo//dp/
HozwdghERFRMbDgJaJcEhISMGzYMHTv3h3Xr19HtWrVsHnzZqxfvx6VK1c2amxCnY30TzcA32yDUGeX+v7c
lHLsbdsEr1d3RdA/
cZh+5ho0vJiNiKhMYcFLRDparRbLly9HvXr1sHbtWkilUrz11lu4cOEC+vTpY+zwntJood55Eth3DtBoDbJ
LhUyK5c3r4uumtbD48m30PnQOSQYotomIqGSw4CUiAMDFixfRoUMHjB8/
HklJSWjWrBmio6Px1VdfwdbW1tjhGZ1EIsHU2lUQ0aYxjj5KRct9J3EpNcPYYRERUSGw4CUq5zIzMzFnzhw
0bdoUBw4cgLW1Nb788ktER0fDx8fH2OGZnK5uFRDdsRkkEsBv30nsvPfI2CEREVEBWPASlWP79u1D06ZN8d
FHHyErKwu9evXChQsXEBwcDAsLjlqYnzp2VjjasRlaOzmg56Fz+OrybU5SQURkwljwEpVDDx8+xOjRo9GpU
ydcunQJ7u7u+OOPP7BlyxZ4eHgYO7wywcHSApsDGuLdulURfOYaxsRcgspAfYqJiKhoWPASlSNCCKxevRr1
6tXDqlWrIJFI8MYbbyA2NhaDBg0q1XFtzZFMIsGnjWtidQsvrL2ViI77z+BeptrYYRER0XNY8BKVE7GxsRj
YPxCjgoLw4MEDNGrUCIcOHcIPP/
wABwcHY4dXpg2v7ob97ZviRkYmWuw9gZjHqcYOiYiInsGCl8iM3blzB1999RX8/f3RsEED/
B0VBaVSiYULFyImJgatWrUydohFp7SEw8YQYNWbpTK1cHH5VbTH8U7NUEmpQJuo0wi7lWjskIiI6D+8KoXI
zDx48AAbNmzA2rVrsX//ft3FVFKpFF1f6YZvvvkadevUMXKUxSeRSCB1tAEkWpPrglHFSoG/
2zfB+JjLGBodi7PJ6fiooSekJhYnEVF5w4KXyAykpKTgzz//
RFhYGHbv3o3s7P+fFKF169YYOnQo+g8YALnSGhUc7IwYqfmzksmwuoUXmjjYYMa56zifkoHVLbxgZ8mPWyI
iY+EnMFEZ9eTJE2zbtg1r167Ftm3boFKpdI81b94cQ4YMweDBg1G9enUAgFYIPE4u+31LhTobGT9sB1RqiL
f6AQrT6daQQyKR4D2vamhob42h0bFoFXUKm1s1RE1bK2OHRkRULrHgJSpD1Go1du/
ejbCwMPz5559IS0vTPVavXj0MHToUgwcPhpeXlxGjLGUaLVR/HXv6/
zdNZLrjfPSq5IRjHZuh7+FzaLHvJNb7N0BHV0djh0VEVO6w4CUycRqNBvv378fatWuxYcMGPHr0/
zN7eXh4YMiQIRg6dCiaNGlicn1aCahvb41jnZph8LGL6HrwDL5pWhuTalU2dlhEROUKC14iEySEwLFjxxAW
FoY//vgDd+/
e1T3m5uaGwYMHY8iQIWjZsiWL3DKgotwSO1o3xrtnr2HyqSs4m5yOr71rQS7lQDlERIbAgpfIRAghcPbsWa
xduxZhYWG4ceOG7rEKFSpgwIABGDp0KNq3bw+ZTGa8QKlYLKQSLGlaC43trfHGySu4kJqB9S3rw0UhN3ZoR
ERmzyROL3z//ffw9PSEUqmEv78/oqOjX7j8kiVL4OXlBSsrK1SrVg1vv/
02MjMzdY8vWrQILVq0gJ2dHVxdXREYGIi4uLjSToOoWC5fvoyPPvoIDRs2RNOmTfHJJ5/
gxo0bsLGxweuvv44tW7bg3r17+Pnnn9GpUycWu2Xc2BqVsK9dE8SmZsBv70mcSU4reCUiInopRj/
DGx4ejuDgYCxduhT+/v5YsmQJunXrhri4OLi6uuZa/
vfff8eMGTOwYsUKBAQE4NKlSxg1ahQkEgkWL14MAPj7778xefJktGjRAtnZ2fjggw/
wyiuv4MKFC7CxsTF0ikS53Lp1C+Hh4Vi7di1OnDiha1coFOjZsyeGDh2KXr16wdra2ohRUmlp7eyA452aod
/h8wjYdwq/
+Hqhg63C2GEREZktoxe8ixcvxvjx4zF69GgAwNKlS7Ft2zasWLECM2bMyLX84cOH0bp1awwbNgwA4OnpiaF
Dh+LYsWO6ZSIiIvTWWbVqFVxdXRETE4N27dqVYjZE+UtMTMT69euxdu1aHDx4UNcuk8nQtWtXDBkyBIGBgZ
zmt5yobq3EwQ7eGP1PHAYeu4gZNd3xcVNbgH2yiYhKnFELXrVajZiYGISEhOjapFIpunTpgiNHjuS5TkBAA
H777TdER0fDz88P165dw/
bt2zFixIh895OcnAwAqFixYp6Pq1QqvTFMU1JSADwdt1T73yxVpSlnJiwhBLSlvrfSZ075vGwuSUlJ2LRpE
8LCwrA3MhJa7dOtSCQStG3bFoOHDMGAAQPg4uKiW6e0XnPmclyEXAa7NcFITcuAkMsM8h4tLVYyKdb61UOj
2HjMvRiPK6psrPCpCxuLstttxVxeZwBzMVXmlAtgXvkYOpeifP4bteB98OABNBoN3Nzc9Nrd3NwQGxub5zr
Dhg3DgwcP0KZNGwghkJ2djYkTJ+KDDz7Ic3mtVou33noLrVu3RqNGjfJcZtGiRZg3b16u9qTkVGiE4c62JK
WYV18+c8qnKLmkp6dj184IbNywAZF7dkOtVuse827WDP0HDERg4KuoXKWKrt2QE0KYxXGxtgSsHZCclmHsS
ErE5MoV4WkhxaQL8Wi99yRWN62BqsqyfTGbWbzO/
sNcTJM55QKYVz6GyiU1pfDfnUbv0lBUUVFRWLhwIX744Qf4+/vjypUrmDZtGj766CPMnj071/KTJ0/
GuXPn9H5Cfl5ISAiCg4N191NSUlCtWjU4OtjB3r70p2EVQiApJQ2O9rZmMcSUOeVT2FxUKhV27tyJ8PBwbP
7rL2Rk/
H8h1rBhQwwZMgSvDR6M2rVrGyLsPJXH41JWCCHQC8BBlwp49egFdP3nMtb710cb57LXvcWcjg1zMU3mlAtg
XvkYOheZpIyc4XV2doZMJkNCQoJee0JCAtzd3fNcZ/
bs2RgxYgTGjRsHAGjcuDHS09MxYcIEzJw5E9JnxrWcMmUKtm7div3796Nq1ar5xqFQKKBQ5L5gRCqRQGqAA
5Zz2l9ioP2VNnPK50W5ZGdnIyoqCmvXrsXGjRuRlJSke6xmzZq6CSHy+2XB0MzluIisbGQs3w2o1MCknpDK
TW9q4aLKOTZNHG1xvFNzDDx6AV0OnMWPzWpjbI1KRo2tqMzldQYwF1NlTrkA5pWPoXMpyj6MWvDK5XL4+Pg
gMjISgYGBAJ52QYiMjMSUKVPyXCcjI0OvqAWgG6bp2b4jb775JjZt2oSoqCjUqFGj9JKgckWr1eLIkSO6CS
ESExN1j1WuXFk3IUSLFi3K/F/qJitbC9Uf//1iM6E7ULZ/+c/
FWWGJXW0bY+qpqxh34jLOJKfjyya1YCHl64mIqLiM3qUhODgYQUFB8PX1hZ+fH5YsWYL09HTdqA
0jR45ElSpVsGjRIgBAnz59sHjxYjRr1kzXpWH27Nno06ePrvCdPHkyfv/9d/
z111+ws7PDvXv3AAAODg6wsrIyTqJUZgkhcPLkSYSHhSE8PBzx8fG6x5ycnDBw4EAMHToUbdq04Ri5VCLkU
imWNq+Dpg42ePP000kqwv3ro6IZnM0mIjIGoxe8gwcPxv379zFnzhzcu3cP3t7eiIiI0F3IFh8fr3dGd9as
WZBIJJg1axbu3LkDFxcX9OnTBwsWLNAt8+OPPwIAOnTooLevlStXYtSoUaWeE5mHrKwsfP7551ixciWuXrm
ia7ezs0NgYCCGDh2KLl26wNKSRQiVjjdqVUY9O2sMPHYBfntPYnNAQzSw51jiRERFJRGiDI/
pU0pSUlLg4OCA5ORk2Nvbl/
r+tELgcXIqKjjYlfn+O4B55JOSkoJBgwZh165dAAClUonevXtjyJAh6NmzZ5n8pcAcjgsAiCdqPOr5dFQVx
21zILMu+xM2FHRsrqU9Qb8j53EzQ4Xf/eqhdyUnI0RZOObyOgOYi6kyp1wA88rH0LkUpV4ziamFiUzJv//
+i3bt2mHXrl2wtrbG4iVf4+69e1i3bh0GDBhQJotdKttq2lrhcAdvdHRxRN/D5/
Fp3C3wXAURUeEZvUsDkSk5f/
48evTogVu3bsHV1RVbtm5FrTpeBhmejuhF7CwtsKlVA8y5cAMzzl3HmeQ0LPepCyv2GyciKhDP8BL9JyoqC
q1bt8atW7dQt25dHDlyBL6+vsYOi0hHKpHg44Y1EOZXD5v+fYh2f5/
GnSeqglckIirnWPASAVi7di26deuG5ORktG7dGocPH0bNmjWNHRblRWEB+9A3ga/
HAIry+SPV4GquONC+Ke5lqtFi70lEP0oxdkhERCaNBS+Va0IIfPrppxg2bBjUajUGDhyIPXv2wMnJdC8KKu
8kUilkNdyA6i6QSMvvR5hPBTsc79QcntZKtPv7NH6LTyh4JSKicqr8fltQuafRaDB58mTMmDEDAPD2228jP
DwcSqXSyJERFY67Uo597ZpgaDVXjDgeh/
fOXoOGF7MREeVSPn8PpHIvPT0dQ4cOxZYtWyCRSPDVV19h2rRpxg6LCkFkZePJb1GASg0xpitQzidjUMikW
OFTF00dbPDOmWs4n5KB3/3qwcGSH+9ERDl4hpfKncTERHTs2BFbtmyBUqnEunXrWOyWJdlaZP66Dwg/
BGRrC16+HJBIJHirTlXsaNMIhx+moOW+k7ic+sTYYRERmQwWvFSuXLp0Ca1atcLx48dRsWJFREZGYsCAAcY
Oi6hEvOJWEcc6ekMIwG/
fSexOeGzskIiITAILXio3Dh8+jICAAFy7dg01atTQ3ScyJ3XtrHGsUzO0qmiH7gfP4uvLdzhJBRGVeyx4qV
zYtGkTOnfujIcPH6JFixY4cuQIvLy8jB0WUalwsLTAltaNEFynKt46cxXjTlyCSsPuH0RUfrHgJbP3zTffY
MCAAcjMzESfPn2wb98+uLm5GTssolIlk0jweZOa+MXXC7/
FJ6LTgTNIyFQbOywiIqNgwUtmS6vV4p133sG0adMghMAbb7yBjRs3wsbGxtihERnMSA83/
N2uKa6lZ8J37wmceJxq7JCIiAyOBS+ZpczMTAwZMgSLFy8GACxatAjff/
89LCw4VBOVPy2d7PFPp2ZwV8rR5u/
T+OP2fWOHRERkUPz2J7Pz6NEj9OvXDwcPHoSlpSVWrVqFYcOGGTssKilyC9j9MBGpaRmAnB9hhVXFSoH97Z
tiXMwlDD52EWeT0zGvgQekEomxQyMiKnX8tiCzcv36dfTo0QNxcXFwcHDApk2b0LFjR2OHRSVIIpPCol5VI
DkVEhl/
pCoKK5kMv7WohyYOtgg5dx1nk9OxuoUX7DhJBRGZOX5bkNn4559/0KpVK8TFxaFatWo4ePAgi12i50gkErz
vVQ2bAxpi7/0kBESdwvV0TlJBROaNBS+Zhe3bt6N9+/
ZISEhA06ZNceTIETRq1MjYYVEpEFnZyAw7AGw6BpGVbexwyqzelZxwtKM3nmi0aLH3JKLuJxk7JCKiUsOCl
8q8n3/+GX379kVGRga6du2K/
fv3o0qVKsYOi0pLthZPlu0Efo3i1MIvqYG9DaI7NYO3gy26HjiLH6/+a+yQiIhKBQteKrOEEJg1axYmTJgA
jUaDUaNGYdu2bbC3tzd2aERlRkW5JXa0aYQ3albCpFNX8MaJy8jS8g8JIjIvvFKByiS1Wo1x48Zh9erVAIC
5c+di7ty5kPCKc6Iis5RK8Y13bTR2sMHkk1dwMTUD61s2gLPC0tihERGVCJ7hpTInOTkZPXv2xOrVqyGTyb
B8+XJ8+OGHLHaJXtL4GpUQ2a4JLqRmoMXeEziTnGbskIiISgQLXipTbt+
+jbZt2yIyMhK2trbYunUrxo4da+ywiMxGW2cHHO/YDA6WFgjYdwp/
3nlg7JCIiF4aC14qM86cOYOWLVvi7NmzcHd3x/
79+9G9e3djh0VkdjxslDjUwRvd3Svi1aMX8PHFmxBCGDssIqJiY8FLZUJkZCTatm2LO3fuoH79+jh69CiaN
Wtm7LCIzJaNhQx/+NfHvAYemH3hJgYfu4j0bI2xwyIiKhaTKHi///57eHp6QqlUwt/
fH9HR0S9cfsmSJfDy8oKVlRWqVauGt99+G5mZmS+1TTJdq1evRvfu3ZGSkoL27dvj0KFD8PDwMHZYZCxyC9
guHgN8NJRTC5cyqUSCOfU9sL5lfWy79wid9p+Blmd6iagMMnrBGx4ejuDgYMydOxcnTpxA06ZN0a1bNyQmJ
ua5/O+//
44ZM2Zg7ty5uHjxIkJDQxEeHo4PPvig2Nsk0ySEwIIFCzBy5EhkZ2djyJAh2LlzJypUqGDs0MiIJDIpLL1r
Ao2qc2phAxlQxQWHO3ijk6sjtKx3iagMMvq3xeLFizF+/
HiMHj0aDRo0wNKlS2FtbY0VK1bkufzhw4fRunVrDBs2DJ6ennjllVcwdOhQvTO4Rd0mmZ7s7Gz873//
w6xZswAA7733HtasWQOFQmHkyIjKp6aOtljUqAYspBwNhYjKHqP+HqhWqxETE4OQkBBdm1QqRZcuXXDkyJE
81wkICMBvv/2G6Oho+Pn54dq1a9i+fTtGjBhR7G2qVCqoVCrd/ZSUFACAVgiD/
HyXczGIEALmMNz7y+aTlpaGIUOGYMf27ZBKpfj6668xafJkADD4z6nmdGzMJReRrYFqy3EgUwXtwNaAZdnv
1mAuxwZgLqaKuZguc8rH0LkUpSYw6jfFgwcPoNFo4Obmptfu5uaG2NjYPNcZNmwYHjx4gDZt2kAIgezsbEy
cOFHXpaE421y0aBHmzZuXqz0pORUaYbizGUkp5jXmZXHySUhIwNDBg3Dm9GlYWVnhp+Wh6NmzFx4np5ZChI
VnTsemzOeSqQa+3QoASO7UCFDKjRxQySnzx+YZzMU0MRfTZU75GCqX1JTC1wZl7tRIVFQUFi5ciB9+
+AH+/v64cuUKpk2bho8+
+gizZ88u1jZDQkIQHBysu5+SkoJq1arB0cEO9vZ2JRV6voQQSEpJg6O9rVlMnlDcfGJjY9GrZ0/
cuHEDLi4u+GvzZvj7+5dipAUzp2NjLrkIuRpJ//
3fwc4WUuuy383FXI4NwFxMFXMxXeaUj6FzkUnKyBleZ2dnyGQyJCQk6LUnJCTA3d09z3Vmz56NESNGYNy4c
QCAxo0bIz09HRMmTMDMmTOLtU2FQpFn31CpRAKpAQ5Yzml/iYH2V9qKk8+BAwfQr18/
PH78GLVr10ZERARq1apVekEWkjkdG3PJRTwTe1nPBQA+/PBD/Pjjj0hMTMSGjRvR/
9VXjR3SSzHl15lEIsGmTZsQGBiY5+M3btxAjRo1cPLkSXh7e2NvVBQ6d+qEh48eoWKFCli1ahXeeustJCUl
Fbiv57dlbKZ8XIrKnHIBzCsfQ+dSlH0Y9aI1uVwOHx8fREZG6tq0Wi0iIyPRqlWrPNfJyMiAVKoftkwmA/
D0L4vibJOMa926dejatSseP36Mli1b4vDhwyZR7BLlGDVqFCQSCSQSCeRyOWrXro358+cjOzv7pbZ78eJFz
Js3Dz8uXYrzsZfQo0ePEorYfN24cQMSiQSnTp0q8W1Xq1YNd+/
eRaNGjfJ8fPDgwbh06VKJbIuIDMvoXRqCg4MRFBQEX19f+Pn5YcmSJUhPT8fo0aMBACNHjkSVKlWwaNEiAE
CfPn2wePFiNGvWTNelYfbs2ejTp4+u8C1om2QahBD46quv8M477wAAAgMDsWbNGlhbWxs5MqLcunfvjpUrV
0KlUmH79u2YPHkyLC0t9S6QLSyNRgOJRIKrV68CAPr164eklLRij0KSlZUFS0vLYq1L/
08mk+X7SyAAWFlZwcrKqkS2RUSGZfRhyQYPHowvvvgCc+bMgbe3N06dOoWIiAjdRWfx8fG4e/
eubvlZs2bhnXfewaxZs9CgQQOMHTsW3bp1w08//
VTobZLxaTQavPXWW7pi980338T69etZ7JLJUigUcHd3h4eHB9544w106dIFmzdvBvB0pJd3330XVapUgY2N
Dfz9/
REVFaVbd9WqVXB0dMTmzZvRoEEDKBQKjBkzBn369AEAWMhkcK7gAODpL1Lz589H1apVoVAo4O3tjYiICN22
cs5whoeHo3379lAqlVizZg1GjRqFwMBALFy4EG5ubnB0dNSdhZ4+fToqVqyIqlWrYuXKlXp5vf/+
+6hbty6sra1Rs2ZNzJ49G1lZWbrHP/zwQ3h7e2P16tXw9PSEg4MDhgwZgtTU/
79YRKvV4rPPPkPt2rVhpVSiaaOGWLhgge7xW7du4bXXXoOjoyMqVqyIfv364caNG/
k+148fP8brr78OFxcXWFlZoU6dOrq4a9SoAQBo1qwZJBIJOnToAAA4fvw4unbtCmdnZzg4OKB9+/
Y4ceJErm3fvXsXPXr0gJWVFWrWrIn169fnem7zO3uccxxzeHp66s78P3vLa1tRUVGQSCSIjIyEr68vrK2tE
RAQgLi4OL19fPzxx3B1dYWdnR3GjRuHGTNmmESXCKIyT1AuycnJAoBITk42yP40Wq148DhZaLRag+yvtBWU
T0ZGhnj11VcFAAFAfPHFF0Jrormb07Exl1y0GSrxoMMH4kGHD0R2eqZB9hkUFCT69eun19a3b1/
RvHlzIYQQ48aNEwEBAWL//
v3iypUr4vPPPxcKhUJcunRJCCHEypUrhaWlpQgICBCHDh0SsbGxIjk5WaxcuVIAEHf+/
Vecj70kNFqtWLx4sbC3txdr164VsbGx4r333hOWlpa6bV2/fl0AEJ6enmLDhg3i2rVr4t9//
xVBQUHCzs5OTJ48WcTGxorQ0FABQHTr1k0sWLBAXLp0SXz00UfC0tJS3Lp1S5fHRx99JA4dOiSuX78uNm/
eLNzc3MSnn36qe3zu3LnC1tZW9O/fX5w9e1bs379fuLu7iw8+
+EC3zHvvvScqVKggVq1aJS5dviy2bo8QPy1bJoQQQq1Wi/
r164sxY8aIM2fOiAsXLohhw4YJLy8voVKp8ny+J0+eLLy9vcXx48fF9evXxe7du8XmzZuFEEJER0cLAGLPn
j3i7t274uHDh0IIISIjI8Xq1avFxYsXxYULF8TYsWOFm5ubSElJ0W0XgHBychI///
yziIuLE7NmzRIymUxcuHBB77k9efLk023u3SsAiIePHumOo4ODg257iYmJ4u7du+Lu3bvi9u3bomXLlqJt2
7Z5bmvfvn0CgPD39xdRUVHi/Pnzom3btiIgIEC3vd9+
+00olUqxYsUKERcXJ+bNmyfs7e1F06ZN83yeisJc3v9CmFcuQphXPobOpSj1GgvePLDgfTkvyuf+/
fuiZcuWAoCQy+UiLCzMCBEWnjkdG3PJRZudLTIPXxQPdp8Q2VlZBtnnswWvVqsVu3fvFgqFQrz77rvi5s2b
QiaTiTt37uit07lzZxESEiKEELrC9tSpU3rLbNq0SQDQOzaVK1cWCxYs0FuuRYsWYtKkSUKI/
y+klixZkitGDw8PodFodG1eXl66AkwIIbKzs4WNjY1Yu3Ztvrl+/vnnwsfHR3d/
7ty5wtraWq9wnD59uvD39xdCCJGSkiIUCoX4+eefhRC5X2erV68WXl5een/
UqlQqYWVlJXbu3JlnDH369BGjR4/O87HnC8n8aDQaYWdnJ7Zs2aJrAyAmTpyot5y/
v79444038tx2QQXvs6ZOnSo8PDxEYmJintvKKXj37NmjW2fbtm0CgHjy5IkulsmTJ+ttt3Xr1ix4n2NOuQh
hXvmYcsFr9D68VH5cvXoVPXr0wOXLl1GhQgX89ddfaNu2rbHDojJGIpPBsqUXkJwKyX/
99g1h69atsLW1RVZWFrRaLYYNG4YPP/wQUVFR0Gg0qFu3rt7yKpUKTk5OuvtyuRxNmjR54T5SUlLw77//
onXr1nrtrVu3xunTp/
XafH19c63fsGFDvYt63dzc9C6akslkcHJy0ptmPTw8HN988w2uXr2KtLQ0ZGdnw97eXm+7np6esLP7/
yEaK1WqpNvGxYsXoVKp0Llz5zxzOn36NK5cuaK3PgBkZmbq+jA/
74033sCAAQNw4sQJvPLKKwgMDERAQECey+ZISEjArFmzEBUVhcTERGg0GmRkZCA+Pl5vuecvXm7VqtVLXwC
3bNkyhIaG4vDhw3BxcXnhss++BipVqgQASExMRPXq1REXF4dJkybpLe/n54e9e/
e+VHxEZAIXrVH5EB0djd69e+P+/fvw8PDAjh07UL9+fWOHRVRoHTt2xI8//
gi5XI7KlSvDwuLpx2daWhpkMhliYmJ0F87msLW11f3fysqqRMeltLGxydX2/
IVrEokkzzat9ungQUeOHMHrr7+OefPmoVu3bnBwcEBYWBi+/PLLArebs42CLuJKS0uDj48P1qxZk+ux/
IrDHj164ObNm9i+fTt2796Nzp07Y/Lkyfjiiy/y3U9QUBAePnyIr7/+Gh4eHlAoFGjVqhXUavUL43tZ+/
btw5tvvom1a9cW+AcNoP9c5rwecp5LIio9Rr9ojczf5s2b0aFDB9y/fx/
NmzfH0aNHWexSsYlsDVQRJ4C9ZyGyNQbbr42NDWrXro3q1avril3g6cVTGo0GiYmJqF27tt6tqFfp29vbo3
Llyjh06JBe+6FDh9CgQYMSyeNZhw8fhoeHB2bOnAlfX1/UqVMHN2/eLNI26tSpAysrK72hIJ/
VvHlzXL58Ga6urrmeHwcHh3y36+LigqCgIPz2229YsmQJli1bBuDpmXLg6YWvzzp06BCmTp2Knj17omHDhl
AoFHjw4EGu7R49ejTX/eJ+Hl25cgUDBw7EBx98gP79+xdrG8/
y8vLC8ePH9dqev09ExcMzvFSqfvjhB7z55pvQarXo0aMH/vjjD72zXkRFlqVBxmcbn/6/
uy9gadyPsbp16+L111/HyJEj8eWXX6JZs2a4f/8+IiMj0aRJE/
Tq1atI25s+fTrmzp2LWrVqwdvbGytXrsSpU6fyPEP6surUqYP4+HiEhYWhRYsW2LZtGzZt2lSkbSiVSrz//
vt47733IJfL0SogANeu30T8zesYP24cXn/9dXz+
+efo16+fbvSJmzdvYuPGjXjvvfdQtWrVXNucM2cOfHx80LBhQ6hUKmzdulVXlLq6usLKygoRERGoWrUqlEo
lHBwcUKdOHaxevRq+vr5ISUnB9OnT8zz7vG7dOvj6+qJNmzZYs2YNoqOjERoaWuTn7smTJ+jTpw+aNWuGCR
Mm4N69e7rHijsc2Ztvvonx48fD19cXAQEBCA8Px5kzZ1CzZs1ibY+I/h/
P8FKp0Gq1CJkxA5MnT4ZWq8W4ceOwefNmFrtkllauXImRI0finXfegZeXFwIDA3H8+HFUr169yNuaOnUqgo
OD8c4776Bx48aIiIjA5s2bUadOnRKPu2/fvnj77bcxZcoUeHt74/
Dhw8Waon327Nl45513MGfOHDRs0ADjxozG/f/6+FpbW2P//v2oXr06+vfvj/
r162Ps2LHIzMzM1Vc4h1wuR0hICJo0aYJ27dpBJpMhLCwMAGBhYYFvvvkGP/30EypXrox+/foBAEJDQ/
H48WM0b94cI0aMwNSpU+Hq6ppr2/
PmzUNYWBiaNGmCX3/9FWvXri3W2fOEhATExsYiMjISlStXRqVKlXS34nr99dcREhKCd999F82bN8f169cxa
tQoKJXKYm+TiJ6SCCEKPxFxOZGSkgIHBwckJyfn+4FckrRC4HFyKio42JX5aQUB4ElmJoYPH4GNG56Ob/
nRRx9h5syZZXKOcHM6NuaSi3iixqOe8wAAjtvmQGZdvMkaTIm5HBuAuZS0rl27wt3dHatXr36p7ZhCLiXFn
HIBzCsfQ+dSlHqNXRqoRKWmpqJv376IioqChYUFQkNDMXLkSGOHRURk8jIyMrB06VJ069YNMpkMa9euxZ49
e7B7925jh0ZU5r1UwXvlyhVcvXoV7dq1g5WVFYQQZfIsHpWcr776ClFRUbC1s8OGDRvwSteuxg6JiKhMkEg
k2L59OxYsWIDMzEx4eXlhw4YN6NKli7FDIyrzilXwPnz4EIMHD8bevXshkUhw+fJl1KxZE2PHjkWFChVyDW
lD5cfGjU8vJlq46BN+SBMRFYGVlRX27Nlj7DCIzFKxLlp7+
+23YWFhgfj4eFhbW+vaBw8erDfnO5Uv169fx+nTpyGTydC9R09jh0NEREQEoJhneHft2oWdO3fmGk6mOGM4
kvnYvHkzAKBt27aoWLGikaMhsyWXwWbOEKRnPAHkhptpjYiIyq5iFbzp6el6Z3ZzPHr0CApF2b9imornzz/
/BPB0qCOi0iKRySDv0AjpBp5amIiIyq5idWlo27Ytfv31V939nGkmP/
vsM3Ts2LHEgqOy49GjRzhw4AAAoO9/42ISERERmYJineH97LPP0LlzZ/
zzzz9Qq9V47733cP78eTx69CjXlJhUPmzbtg0ajQZNmjRBjRo18Dg51dghkZkSGg3U+y8AGU8gujUHLDi6I
hERvVixzvA2atQIly5dQps2bdCvXz+kp6ejf//+OHnyJGrVqlXSMVIZkNOdoR/
P7lJpU2uQPj8M+OIvQK0xdjRERFQGFPnUSFZWFrp3746lS5di5syZpRETlTFPnjzBzp07AQCBgYHGDYaIiI
joOUU+w2tpaYkzZ86URixURkVGRiI9PR3VqlVDs2bNjB0OERERkZ5idWkYPnw4QkNDSzoWKqP+
+usvAE9HZ+BMe0RERGRqinW1R3Z2NlasWIE9e/
bAx8cHNjY2eo8vXry4RIIj06fRaHTj77I7AxEREZmiYhW8586dQ/PmzQEAly5d0nuMZ/
jKl2PHjiExMREODg5o3769scMhIiIiyqVYBe++fftKOg4qo3K6M/
Tq1QuWlpZGjoaIiIgot5cewPL27dsAkGuaYSofOBwZGZylDNbv9UfGk0zAkjOtERFRwYp10ZpWq8X8+fPh4
OAADw8PeHh4wNHRER999BG0Wm1Jx0gmKjY2FpcuXYJcLkf37t2NHQ6VExILGRTdmwOdGkNiwYKXiIgKVqwz
vDNnzkRoaCg++eQTtG7dGgBw8OBBfPjhh8jMzMSCBQtKNEgyTTlndzt16gR7e3vjBkNERESUj2IVvL/
88guWL1+Ovn376tqaNGmCKlWqYNKkSSx4y4mc/
rvszkCGJDQaZEVfBtKfQHRozKmFiYioQMXq0vDo0SPUq1cvV3u9evXw6NGjIm/v+++/h6enJ5RKJfz9/
REdHZ3vsh06dIBEIsl169Wrl26ZtLQ0TJkyBVWrVoWVlRUaNGiApUuXFjkuyt/
du3dx7NgxAND7w4eo1Kk1SPtgNbBgPacWJiKiQilWwdu0aVN89913udq/+
+47NG3atEjbCg8PR3BwMObOnYsTJ06gadOm6NatGxITE/
NcfuPGjbh7967udu7cOchkMgwaNEi3THBwMCIiIvDbb7/h4sWLeOuttzBlyhTde
LH08rZs2QIhBPz8/FC5cmVjh0NERESUr2L9FvjZZ5+hV69e2LNnD1q1agUAOHLkCG7duoXt27cXaVuLFy/
G+PHjMXr0aADA0qVLsW3bNqxYsQIzZszItXzFihX17oeFhcHa2lqv4D18+DCCgoLQoUMHAMCECRPw008/
ITo6mmcjS0hOdwZONkFERESmrlgFb/v27REXF4cffvgBsbGxAID+/
ftj0qRJRTrbp1arERMTg5CQEF2bVCpFly5dcOTIkUJtIzQ0FEOGDNGb7S0gIACbN2/
GmDFjULlyZURFReHSpUv46quv8tyGSqWCSqXS3U9JSQEAaIWAVohC51Nc4r99CCFQFsa4SE1NxZ49ewAAff
r2zfUclbV8XoS5mB7xzOtNGOg9WtrM5dgAzMVUMRfTZU75GDqXonz+F/
tqjypVqrz0xWkPHjyARqOBm5ubXrubm5uukH6R6OhonDt3DqGhoXrt3377LSZMmICqVavCwsICUqkUP//
8M9q1a5fndhYtWoR58+blak9KToVGGG7muKSUNIPt62Vs/
utPqNVq1KhZE+6Vq+Jxcmqey5WVfAqDuZiQTLXuv8mpaUCW+gULly1l/tg8g7mYJuZiuswpH0PlkpqSd/
2Rl2IVvCtXroStra1eNwIAWLduHTIyMhAUFFSczRZZaGgoGjduDD8/P732b7/9FkePHsXmzZvh4eGB/
fv3Y/LkyahcuTK6dOmSazshISEIDg7W3U9JSUG1atXg6GAHe3u7Us9DCIGklDQ42tuWiamZ9+zeBQDo/
+qrqOiYeziyspbPizAX0yPkaiT9938HO1tIrRXGDKdEmMuxAZiLqWIupsuc8jF0LjJJKZ/
hXbRoEX766adc7a6urpgwYUKhC15nZ2fIZDIkJCTotSckJMDd3f2F66anpyMsLAzz58/Xa3/y5Ak+
+OADbNq0STdyQ5MmTXDq1Cl88cUXeRa8CoUCCkXuL02pRAKpAQ5Yzml/iYH29zKysrKwfds2AE/
77+YVb1nKpyDMxfSIZ2Iv67nkMJdjAzAXU8VcTJc55WPoXIqyj2KN0hAfH48aNWrkavfw8EB8fHyhtyOXy+
Hj44PIyEhdm1arRWRkpO5iuPysW7cOKpUKw4cP12vPyspCVlYWpFL91GQyGWeBKwEHDhxAUlISXFxcCjxGR
KXCUgarqb2B8V05tTARERVKsc7wurq64syZM/
D09NRrP336NJycnIq0reDgYAQFBcHX1xd+fn5YsmQJ0tPTdaM2jBw5ElWqVMGiRYv01gsNDUVgYGCu/
dnb26N9+/aYPn06rKys4OHhgb///hu//vorFi9eXPRkSU/
O7Gp9+vSBTMZigwxPYiGDMrAlniSncmphIiIqlGIVvEOHDsXUqVNhZ2enuxDs77//
xrRp0zBkyJAibWvw4MG4f/
8+5syZg3v37sHb2xsRERG6C9ni4+Nzna2Ni4vDwYMHsWvXrjy3GRYWhpCQELz+
+ut49OgRPDw8sGDBAkycOLEY2VIOIQSHIyMiIqIyRyJE0cf0UavVGDFiBNatWweL/
6b11Gq1GDlyJJYuXQq5XF7igRpSSkoKHBwckJycDHv73BdllTStEHicnIoKDnYm3X/
n5MmTaN68OaytrfHgwQNYWVnluVxZyacwmIvpERot1GeuIy39CRxb1ofMDM7ymsuxAZiLqWIupsuc8jF0Lk
Wp14p1hlculyM8PBwff/wxTp06BSsrKzRu3BgeHh7FCpjKhpyzu6+88kq+xS5RqVNnIy14xdP/
b5sDmEHBS0REpavY4/
ACQJ06dVCnTh1oNBqcPXsW9vb2qFChQknFRiaG3RmIiIioLCrWKA1vvfWWbrIHjUaD9u3bo3nz5qhWrRqio
qJKMj4yETdu3MCpU6cglUp1w70RERERlQXFKnjXr1+Ppk2bAgC2bNmCa9euITY2Fm+//
TZmzpxZogGSadi8eTMAoG3btnB2djZyNERERESFV6yC98GDB7qJIbZv347XXnsNdevWxZgxY3D27NkSDZBM
Q85wZP369TNuIERERERFVKyC183NDRcuXIBGo0FERAS6du0KAMjIyODYrGbo0aNH2L9/
PwAWvERERFT2FOuitdGjR+O1115DpUqVIJFIdNP1Hjt2DPXq1SvRAMn4tm/
fDo1Gg8aNG6NmzZrGDoeIiIioSIpV8H744Ydo1KgRbt26hUGDBkGhUAB4On3vjBkzSjRAMj52ZyCTYiGF1Y
RueJKpAiyK9SMVERGVM8UelmzgwIEAgNu3b0Or1UIqlSIoKKjEAiPTkJmZiYiICAAcjoxMg8TSAsohbZ9OL
Wz5UiMrEhFROfHSp0caNGiAGzdulEAoZIoiIyORnp6OqlWronnz5sYOh4iIiKjIXrrgLcbMxFSG5Ew20a9f
P0jK+JSHZB6ERovs2NvA5bsQGq2xwyEiojKAvwdSvrRarW78XfbfJZOhzkbqpKVP/
8+phYmIqBBe+gzvBx98gIoVK5ZELGRijh07hoSEBNjb26N9+/
bGDoeIiIioWF76DG9ISEhJxEEmKKc7Q69evSCXy40cDREREVHxlOiYPrdu3cKYMWNKcpNkRByOjIiIiMxBi
Ra8jx49wi+//
FKSmyQjiY2NRVxcHCwtLdGjRw9jh0NERERUbEXq0pBzAVN+rl279lLBkOnI6c7QqVMn2NvbGzkaIiIiouIr
UsEbGBgIiUTywqHIOHSVeXh2ODIiIiKisqxIXRoqVaqEjRs3QqvV5nk7ceJEacVJBnTv3j0cPXoUANC3b18
jR0P0HAsplCM7AoNbc2phIiIqlCJ9W/
j4+CAmJibfxws6+0tlw5YtWyCEQIsWLVClShVjh0OkR2JpAatRnYEhbTi1MBERFUqRvi2mT5+O9PT0fB+vX
bs29u3b99JBkXHldGcIDAw0biBEREREJaBIBW+VKlVQo0aNfB+3sbHhBAVlXFpaGvbs2QOA/
XfJNAmtFpobiUBaOkRDG0DGmdaIiOjFitSloU6dOrh//77u/
uDBg5GQkFDiQZHx7Ny5EyqVCrVr10aDBg2MHQ5RbqpspIz9Fpi2AlBlGzsaIiIqA4pU8D7fP3f79u0v7OJA
Zc+zk01wxA0iIiIyB7zEmXSysrKwbds2AOzOQEREROajSAWvRCLJddaPZwHNx8GDB/
H48WM4OzsjICDA2OEQERERlYgid2kYNWoU+vfvj/
79+yMzMxMTJ07U3c+5FdX3338PT09PKJVK+Pv7Izo6Ot9lO3TooCu8n7316tVLb7mLFy+ib9+
+cHBwgI2NDVq0aIH4+Pgix1ae5HRn6NOnD2S8EIiIiIjMRJFGaQgKCtK7P3z48JcOIDw8HMHBwVi6dCn8/
f2xZMkSdOvWDXFxcXB1dc21/MaNG6FWq3X3Hz58iKZNm2LQoEG6tqtXr6JNmzYYO3Ys5s2bB3t7e5w/
fx5KpfKl4zVXQggOR0ZERERmqUgF78qVK0s8gMWLF2P8+PEYPXo0AGDp0qXYtm0bVqxYgRkzZuRavmLFinr
3w8LCYG1trVfwzpw5Ez179sRnn32ma6tVq1aJx25OTp8+jZs3b8LKygpdunQxdjhEREREJcao0xSp1WrExM
QgJCRE1yaVStGlSxccOXKkUNsIDQ3FkCFDYGNjAwDQarXYtm0b3nvvPXTr1g0nT55EjRo1EBISku+ZS5VKB
ZVKpbufkpLydFtCQGuAmeNyRr8QQkBb6nvLW053hq6vvAKlldVL5W0K+ZQU5mJ6hEwCxWutoVJlQcgkBnmP
ljZzOTYAczFVzMV0mVM+hs6lKJ//
Ri14Hzx4AI1GAzc3N712Nzc3xMbGFrh+dHQ0zp07h9DQUF1bYmIi0tLS8Mknn+Djjz/
Gp59+ioiICPTv3x/79u3Lc2KMRYsWYd68ebnak5JToRGGuygvKSXNYPt63sZNmwAAXV/
phsfJqSWyTWPmU9KYi4kZ2gYAkPwkE3iSaeRgSo5ZHJv/
MBfTxFxMlznlY6hcUlMKX6+U6YnoQ0ND0bhxY/j5+enatNqnf1P069cPb7/9NgDA29sbhw8fxtKlS/
MseENCQhAcHKy7n5KSgmrVqsHRwQ729nalnMXTv4SSUtLgaG9rlFEvbt68ibNnzkAqleK1QQNRweHlcjZ2P
iWJuZgmc8oFMK98mItpYi6my5zyMXQuMkkZOcPr7OwMmUyWa7a2hIQEuLu7v3Dd9PR0hIWFYf78+bm2aWFh
kWuWsPr16+PgwYN5bkuhUEChUORql0okkBrggOWc9pcYaH/P27plCwCgTZs2cHVxeentGTufksRcTI/
QaqFJSAJS0wE7G0ilZX84cXM5NgBzMVXMxXSZUz6GzqUo+zDqN4VcLoePjw8iIyN1bVqtFpGRkWjVqtUL11
23bh1UKlWukSLkcjlatGiBuLg4vfZLly7Bw8Oj5II3I8/OrkZk8lTZSBn2JfC/
pZxamIiICsXoXRqCg4MRFBQEX19f+Pn5YcmSJUhPT9eN2jBy5EhUqVIFixYt0lsvNDQUgYGBcHJyyrXN6dO
nY/DgwWjXrh06duyIiIgIbNmyBVFRUYZIqUx5/Pgx/
v77bwAseImIiMg8Gb3gHTx4MO7fv485c+bg3r178Pb2RkREhO5Ctvj4+Fw/
WcbFxeHgwYPYtWtXntt89dVXsXTpUixatAhTp06Fl5cXNmzYgDZt2pR6PmXNtm3boNFo0KhRIw7dRkRERGb
J6AUvAEyZMgVTpkzJ87G8zsp6eXnphr7Iz5gxYzBmzJiSCM+s5Uw2wbO7REREZK7K/
tUeVGyZmZmIiIgAwNnViIiIyHyx4C3H9u7di7S0NFSpUgU+Pj7GDoeIiIioVLDgLcee7c5Q1sf+IyIiIsqP
SfThJcPTarXYvHkzAPbfpTJGJoWinz9UKjUg49/
sRERUMBa85VR0dDTu3bsHe3t7dOjQwdjhEBWaRG4B62l9oEpOhUTOjzAiIioYT4+UUzndGXr27Am5XG7kaI
iIiIhKDwvecoqzq1FZJYSANikdSM4ocHhCIiIigAVvuRQXF4fY2FhYWlqiR48exg6HqGgys5DcfxEw6lsgM
8vY0RARURnAgrccyunO0LFjRzg4OBg5GiIiIqLSxYK3HMopeDnZBBEREZUHLHjLmYSEBBw5cgQA0LdvXyNH
Q0RERFT6WPCWM1u2bIEQAr6+vqhSpYqxwyEiIiIqdSx4yxl2ZyAiIqLyhgVvOZKWlobdu3cD4HBkREREVH5
wmqJyZNeuXVCpVKhVqxYaNmxo7HCIikcmhbxbM6jVWZxamIiICoUFbzny7GQTEonEuMEQFZNEbgGb9wdAza
mFiYiokHh6pJzIzs7Gtm3bALA7AxEREZUvLHjLiYMHD+LRo0dwdnZGQECAscMhKjYhBMQTNZCp5tTCRERUK
Cx4y4mc7gy9e/eGhQV/BqYyLDMLSb3mA0O/
4tTCRERUKCx4ywEhBIcjIyIionKLBW85cObMGdy4cQNWVlbo2rWrscMhIiIiMigWvOVAztndrl27wtra2sj
REBERERkWC95yIKf/
LrszEBERUXnEgtfMxcfH4+TJk5BKpejdu7exwyEiIiIyOBa8Zm7z5s0AgNatW8PFxcXI0RAREREZHsenMnP
Pzq5GZBZkEli2a4isrGxAxhkDiYioYCZxhvf777+Hp6cnlEol/
P39ER0dne+yHTp0gEQiyXXr1atXnstPnDgREokES5YsKaXoTdfjx4/x999/
A2DBS+ZDIreE7YdDgfcCIZFbGjscIiIqA4xe8IaHhyM4OBhz587FiRMn0LRpU3Tr1g2JiYl5Lr9x40bcvXt
Xdzt37hxkMhkGDRqUa9lNmzbh6NGjqFy5cmmnYZK2b9+O7OxsNGzYELVr1zZ2OERERERGYfSCd/HixRg/
fjxGjx6NBg0aYOnSpbC2tsaKFSvyXL5ixYpwd3fX3Xbv3g1ra+tcBe+dO3fw5ptvYs2aNbC0LJ9ngXKGI+P
ZXSIiIirPjNqHV61WIyYmBiEhIbo2qVSKLl264MiRI4XaRmhoKIYMGQIbGxtdm1arxYgRIzB9+nQ0bNiwwG
2oVCqoVCrd/ZSUlKfbEQJaIQqbTrGJ//
YhhIC2hLapUqmwY8cOAEDffv0MkkeO0sjHWJiL6RFP1E+nFgag3TobsFYYOaKXZy7HBmAupoq5mC5zysfQu
RSltjFqwfvgwQNoNBq4ubnptbu5uSE2NrbA9aOjo3Hu3DmEhobqtX/66aewsLDA1KlTCxXHokWLMG/
evFztScmp0AjDXRSTlJJWYtvas3s30tLS4F6pEmrWrovHyakltu3CKsl8jI25mJBMte6/
yalpQJb6BQuXLWX+2DyDuZgm5mK6zCkfQ+WSmlL42qZMj9IQGhqKxo0bw8/
PT9cWExODr7/+GidOnIBEUrhiNSQkBMHBwbr7KSkpqFatGhwd7GBvb1ficT9PCIGklDQ42tsWOuaC7N2zCw
AQ2K8fnCo4lMg2C6s08jEW5mJ6hFyNpP/
+72BnC6mZnOE1h2MDMBdTxVxMlznlY+hcZJIycobX2dkZMpkMCQkJeu0JCQlwd3d/4brp6ekICwvD/
Pnz9doPHDiAxMREVK9eXdem0WjwzjvvYMmSJbhx40aubSkUCigUub80pRIJpAY4YDmn/
SUltD+tVqsbfzcwMNAgOejt/79/
SyofY2Iupkc8E3tZzyWHuRwbgLmYKuZiuswpH0PnUpR9GPWiNblcDh8fH0RGRuratFotIiMj0apVqxeuu27
dOqhUKgwfPlyvfcSIEThz5gxOnTqlu1WuXBnTp0/
Hzp07SyUPU3P8+HHcu3cPdnZ26NChg7HDISIiIjIqo3dpCA4ORlBQEHx9feHn54clS5YgPT0do0ePBgCMHD
kSVapUwaJFi/
TWCw0NRWBgIJycnPTanZyccrVZWlrC3d0dXl5epZuMiciZbKJnz555nrkmIiIiKk+MXvAOHjwY9+/
fx5w5c3Dv3j14e3sjIiJCdyFbfHw8pFL9E9FxcXE4ePAgdu3aZYyQTR6HIyMiIiL6f0YveAFgypQpmDJlSp
6PRUVF5Wrz8vLSDX1RGHn12zVXly5dwsWLF2FpaYmePXsaOxyikieTwMK/
LrKzObUwEREVjkkUvFRycs7udujQAQ4Ohh2dgcgQJHJL2C0aicfJqZxamIiICsXoM61RycopeAMDA40bCBE
REZGJYMFrRhISEnD48GEAQN+
+fY0cDREREZFpYMFrRrZu3QohBHx8fFC1alVjh0NUKsQTNR73nAcMWQzxxHxmWSMiotLDPrxmhN0ZqNzIzD
J2BEREVIbwDK+ZSE9Px+7duwFwODIiIiKiZ7HgNRO7du1CZmYmatasiUaNGhk7HCIiIiKTwYLXTOTMrtavX
z9Iyvhc3EREREQliQWvGcjOzsbWrVsBsDsDERER0fNY8JqBQ4cO4dGjR3ByckLr1q2NHQ4RERGRSeEoDWYg
pztD7969YWHBQ0pmTiqBRVNPZGdrACm77xARUcFYHZVxQggOR0blikRhCbuvxj2dWljBqYWJiKhg7NJQxp0
9exbXr1+HUqlE165djR0OERERkclhwVvG5ZzdfeWVV2BjY2PkaIiIiIhMDwveMu7Z4ciIygPxRI2kVxcCQd
9wamEiIioU9uEtw27duoUTJ05AIpGgd+/
exg6HyGBEcoaxQyAiojKEZ3jLsM2bNwMAWrduDVdXVyNHQ0RERGSaWPCWYezOQERERFQwFrxlVFJSEqKiog
Cw4CUiIiJ6ERa8ZdT27duRnZ2NBg0aoE6dOsYOh4iIiMhkseAto3KGI+PZXSIiIqIX4ygNZZBKpcKOHTsAc
HY1KoekEsi8qkCj4dTCRERUOCx4y6B9+/YhNTUVlSpVgq+vr7HDITIoicIS9j+
+wamFiYio0NiloQx6tjuDVMpDSERERPQirJbKGK1Wy/
67REREREXAgreM+eeff3D37l3Y2dmhY8eOxg6HyOBEphrJQ78AJvwIkcmphYmIqGAmUfB+//
338PT0hFKphL+/P6Kjo/NdtkOHDpBIJLluvXr1AgBkZWXh/
fffR+PGjWFjY4PKlStj5MiR+Pfffw2VTqnKmWyiR48eUCgUxg2GyBgEoE1IAu6nAMLYwRARUVlg9II3PDwc
wcHBmDt3Lk6cOIGmTZuiW7duSExMzHP5jRs34u7du7rbuXPnIJPJMGjQIABARkYGTpw4gdmzZ+PEiRPYuHE
j4uLi0LdvX0OmVWrYnYGIiIioaIw+SsPixYsxfvx4jB49GgCwdOlSbNu2DStWrMCMGTNyLV+xYkW9+2FhYb
C2ttYVvA4ODti9e7feMt999x38/PwQHx+P6tWrl1Impe/
y5cu4cOECLCws0LNnT2OHQ0RERFQmGLXgVavViImJQUhIiK5NKpWiS5cuOHLkSKG2ERoaiiFDhsDGxibfZZ
KTkyGRSODo6Jjn4yqVCiqVSnc/JSUFAKAVAlpR+r+Ziv/
2IYSA9gXL5XRnaN+hA+wdHAwSW3EUNp+ygLmYHvHM614Y6D1a2szl2ADMxVQxF9NlTvkYOpeifP4bteB98O
ABNBoN3Nzc9Nrd3NwQGxtb4PrR0dE4d+4cQkND810mMzMT77//PoYOHQp7e/
s8l1m0aBHmzZuXqz0pORUaYbiB7ZNS0l74+MaNmwAAr7zSHY+TUw0R0kspKJ+yhLmYkGcuVEtOTQOyzOfCt
TJ/bJ7BXEwTczFd5pSPoXJJTSl8LWT0Lg0vIzQ0FI0bN4afn1+ej2dlZeG1116DEAI//
vhjvtsJCQlBcHCw7n5KSgqqVasGRwc72NvblXjczxNCICklDY72tpBI8i6wExMTcezYUQDA4MGDUMGh9OMq
rsLkU1YwF9Mj5Gok/fd/BztbSK3L/
sWb5nJsAOZiqpiL6TKnfAydi0xSRs7wOjs7QyaTISEhQa89ISEB7u7uL1w3PT0dYWFhmD9/fp6P5xS7N2/
exN69e/M9uwsACoUizxEPpBIJpAY4YDmn/SUv2N/
2bdsghEDz5s3hYeL9kAuTT1nBXEyPkEog9XCFVquBRFq2c8lhLscGYC6mirmYLnPKx9C5FGUfRh2lQS6Xw8
fHB5GRkbo2rVaLyMhItGrV6oXrrlu3DiqVCsOHD8/1WE6xe/
nyZezZswdOTk4lHruh5YzOEBgYaNxAiIxMopTDYeVU4JtxkCjlxg6HiIjKAKN3aQgODkZQUBB8fX3h5+eHJ
UuWID09XTdqw8iRI1GlShUsWrRIb73Q0FAEBgbmKmazsrIwcOBAnDhxAlu3boVGo8G9e/
cAPB3hQS4ve1+Q6enp2LVrFwAOR0ZERERUVEYveAcPHoz79+9jzpw5uHfvHry9vREREaG7kC0+Ph5Sqf6J6
Li4OBw8eFBXBD7rzp072Lx5MwDA29tb77F9+/
ahQ4cOpZJHadq9ezcyMzNRo0YNNG7c2NjhEBEREZUpRi94AWDKlCmYMmVKno9FRUXlavPy8tIbmuhZnp6e+
T5WVuUMR9avX78y36Gd6GWJTDWSJ/4IaDUQP00GrMr+RWtERFS6TKLgpfxlZ2dj69atANh/
lwjA06mFbybq/
k9ERFQQo08tTC926NAhPHz4EBUrVkTr1q2NHQ4RERFRmcOC18TljM7Qu3dvWFjwhDwRERFRUbHgNWFCCA5H
RkRERPSSWPCasHPnzuHatWtQKpV45ZVXjB0OERERUZnEgteE5Zzd7dq1K2xsbIwcDREREVHZxE6hJuzZ4ci
I6D8SQOrmCK1WC3CUPiIiKgQWvCbq9u3biImJgUQiQe/
evY0dDpHJkCjlcFj7Lh4np3JqYSIiKhR2aTBRObPFBQQE6GadIyIiIqKiY8FrotidgYiIiKhksOA1QUlJSd
i3bx8ADkdG9DyhykLKGz8C03+BUGUZOxwiIioD2IfXBO3YsQPZ2dmoX78+6tSpY+xwiEyLVkATd0f3fyIio
oLwDK8J4mQTRERERCWHBa+JUalU2L59OwD23yUiIiIqCSx4TUxUVBRSU1NR
qVIltGjRwtjhEBEREZV5LHhNTE53hr59+0Iq5eEhIiIielmsqEyIVqvVFbzszkBERERUMjhKgwmJiYnBv//
+C1tbW3Tq1MnY4RCZLImDNYTgCA1ERFQ4LHhNyF//
TTbRo0cPKBQK4wZDZKIkVnI4bvrg6dTCVpxamIiICsYuDSYkZzphdmcgIiIiKjkseE3EtWtXcf78eVhYWKB
nz57GDoeIiIjIbLDgNRE7/
ht7t3379qhQoYKRoyEyXUKVhdS3lwOzfufUwkREVCjsw2sicgpezq5GVACtQPbpG7r/ExERFYRneE3A/
fv3EX3sKICn4+8SERERUclhwWsCtm7dCq1Wi2bNmqF69erGDoeIiIjIrJhEwfv999/D09MTSqUS/
v7+iI6OznfZDh06QCKR5Lr16tVLt4wQAnPmzEGlSpVgZWWFLl264PLly4ZIpVg42QQRERFR6TF6wRseHo7g
4GDMnTsXJ06cQNOmTdGtWzckJibmufzGjRtx9+5d3e3cuXOQyWQYNGiQbpnPPvsM33zzDZYuXYpjx47BxsY
G3bp1Q2ZmpqHSKpJGDRvCw8MDfVnwEhEREZU4oxe8ixcvxvjx4zF69Gg0aNAAS5cuhbW1NVasWJHn8hUrVo
S7u7vutnv3blhbW+sKXiEElixZglmzZqFfv35o0qQJfv31V/
z777/487+JHUzNxwsW4J+Tp9GkSRNjh0JERERkdow6SoNarUZMTAxCQkJ0bVKpFF26dMGRI0cKtY3Q0FAMG
TIENjY2AIDr16/
j3r176NKli24ZBwcH+Pv748iRIxgyZEiubahUKqhUKt39lJQUAIBWCGgNMH2pEAISiUS3z7IuZ8pXIQS0Ro
7lZTEX0yOEAJSWgPgvF75nTApzMU3MxXSZUz6GzqUon/
9GLXgfPHgAjUYDNzc3vXY3NzfExsYWuH50dDTOnTuH0NBQXdu9e/d023h+mzmPPW/
RokWYN29ervak5FRohKTAOEpKUkqawfZlCOaUD3MxMWuDAQDJWWogWW3kYEqOWRyb/
zAX08RcTJc55WOoXFJTUgu9bJkehzc0NBSNGzeGn5/
fS20nJCQEwcHBuvspKSmoVq0aHB3sYG9v97JhFkgIgaSUNDja2+rO9JZl5pQPczFN5pQLYF75MBfTxFxMlz
nlY+hcZJIycobX2dkZMpkMCQkJeu0JCQlwd3d/4brp6ekICwvD/
Pnz9dpz1ktISEClSpX0tunt7Z3nthQKBRQKRa52qUQCqQEOWM5pf4mB9lfazCkf5mKazCkXwLzyYS6mibmY
LnPKx9C5FGUfRr1oTS6Xw8fHB5GRkbo2rVaLyMhItGrV6oXrrlu3DiqVCsOHD9drr1GjBtzd3fW2mZKSgmP
HjhW4TSIyfUKdhdSQX4GP10GoObUwEREVzOhdGoKDgxEUFARfX1/4+flhyZIlSE9Px+jRowEAI0eORJUqVb
Bo0SK99UJDQxEYGAgnJye9dolEgrfeegsff/wx6tSpgxo1amD27NmoXLkyp+0lMgcagexjl3T/
JyIiKojRC97Bgwfj/v37mDNnDu7duwdvb29EREToLjqLj4+HVKp/IjouLg4HDx7Erl278tzme++9h/
T0dEyYMAFJSUlo06YNIiIioFQqSz0fIiIiIjItRi94AWDKlCmYMmVKno9FRUXlavPy8tINfZEXiUSC+fPn5
+rfS0RERETlj9EnniAiIiIiKk0seImIiIjIrLHgJSIiIiKzZhJ9eE1NTv/
gnCmGS5tWCKSmpEImEWV+DD7AvPJhLqZHPFEjNfvpVODSlBTIsnOPoV3WmMuxAZiLqWIupsuc8jF0Ljl12o
uu68ohEYVZqpy5ffs2qlWrZuwwiIiIiKgAt27dQtWqVV+4DAvePGi1Wvz777+ws7MzyNR4OVMZ37p1C/
b29qW+v9JmTvkwF9NkTrkA5pUPczFNzMV0mVM+hs5FCIHU1FRUrlw51xC2z2OXhjxIpdIC/
1IoDfb29mX+xf4sc8qHuZgmc8oFMK98mItpYi6my5zyMWQuDg4OhVqOF60RERERkVljwUtEREREZo0FrwlQ
KBSYO3cuFIqyf7U5YF75MBfTZE65AOaVD3MxTczFdJlTPqacCy9aIyIiIiKzxjO8RERERGTWWPASERERkVl
jwUtEREREZo0FLxERERGViKioKEgkEiQlJQEAVq1aBUdHR6PGBLDgLTGjRo2CRCLBJ598otf+559/
GmS2NkP6/vvv4enpCaVSCX9/f0RHR79w+QULFiAgIADW1tYm8aJ/
XlHyuXHjBsaOHYsaNWrAysoKtWrVwty5c6FWqw0Ycf6Kemz69u2L6tWrQ6lUolKlShgxYgT+/
fdfA0X7YkXNJYdKpYK3tzckEglOnTpVukEWUlFz8fT0hEQi0bs9/9liLMU5Ltu2bYO/
vz+srKxQoUIFBAYGln6ghVSUfHK+yPO6HT9+3IBR562ox+bSpUvo168fnJ2dYW9vjzZt2mDfvn0GivbFipr
LiRMn0LVrVzg6OsLJyQkTJkxAWlqagaLVVxrfkfHx8ejVqxesra3h6uqK6dOnIzs7uxSiL7wjR45AJpOhV6
9eRo2j0ASViKCgIKFUKoWjo6N49OiRrn3Tpk3CnJ7msLAwIZfLxYoVK8T58+fF+PHjhaOjo0hISMh3nTlz5
ojFixeL4OBg4eDgYLhgC6Go+ezYsUOMGjVK7Ny5U1y9elX89ddfwtXVVbzzzjsGjjy34hybxYsXiyNHjogb
N26IQ4cOiVatWolWrVoZMOq8FSeXHFOnThU9evQQAMTJkydLP9gCFCcXDw8PMX/
+fHH37l3dLS0tzYBR5604uaxfv15UqFBB/PjjjyIuLk6cP39ehIeHGzDq/
BU1H5VKpXdM7t69K8aNGydq1KghtFqtgaPXV5xjU6dOHdGzZ09x+vRpcenSJTFp0iRhbW0t7t69a8DIcytq
Lnfu3BEVKlQQEydOFLGxsSI6OloEBASIAQMGGDjy0vmOzM7OFo0aNRJdunQRJ0+eFNu3bxfOzs4iJCSkFDM
p2NixY8W0adOEra2tuHPnjq593759AoB4/PixEEKIlStXmsR3v/lUYkYWFBQkevfuLerVqyemT5+ua3+
+4F2/fr1o0KCBkMvlwsPDQ3zxxRd62/
Hw8BALFiwQo0ePFra2tqJatWrip59+0lsmPj5eDBo0SDg4OIgKFSqIvn37iuvXr5dqfjn8/
PzE5MmTdfc1Go2oXLmyWLRoUYHrmsqL/
lkvk0+Ozz77TNSoUaM0wiuSksjlr7/+EhKJRKjV6tIIsdCKm8v27dtFvXr1xPnz502m4C1OLh4eHuKrr74y
QHRFU9RcsrKyRJUqVcTy5csNFWKRvOx7Rq1WCxcXFzF//vzSCrHQiprL/fv3BQCxf/
9+XVtKSooAIHbv3l3q8b5IUXP56aefhKurq9BoNLq2M2fOCADi8uXLpR7vs0rjO3L79u1CKpWKe/
fu6dp+/
PFHYW9vL1QqVYnEXVSpqanC1tZWxMbGisGDB4sFCxboHjPVgpddGkqQTCbDwoUL8e233+L27du5Ho+JicFr
r72GIUOG4OzZs/jwww8xe/
ZsrFq1Sm+5L7/8Er6+vjh58iQmTZqEN954A3FxcQCArKwsdOvWDXZ2djhw4AAOHToEW1tbdO/evdR/
Vler1YiJiUGXLl10bVKpFF26dMGRI0dKdd+loaTySU5ORsWKFUsjxEIriVwePXqENWvWICAgAJaWlqUVaoG
Km0tCQgLGjx+P1atXw9ra2hChFuhljssnn3wCJycnNGvWDJ9//
rnRf74sTi4nTpzAnTt3IJVK0axZM1SqVAk9evTAuXPnDBV2vkriPbN582Y8fPgQo0ePLq0wC6U4uTg5OcHL
ywu//vor0tPTkZ2djZ9++gmurq7w8fExVOi5FCcXlUoFuVwOqfT/
SxorKysAwMGDB0s34GeU1nfkkSNH0LhxY7i5uenaunXrhpSUFJw/f/6lYi6uP/74A/
Xq1YOXlxeGDx+OFStWQJj4tA4seEvYq6++Cm9vb8ydOzfXY4sXL0bnzp0xe/
Zs1K1bF6NGjcKUKVPw+eef6y3Xs2dPTJo0CbVr18b7778PZ2dnXb+q8PBwaLVaLF+
+HI0bN0b9+vWxcuVKxMfHIyoqqlRze/DgATQajd6bDgDc3Nxw7969Ut13aSiJfK5cuYJvv/0W//vf/
0ojxEJ7mVzef/
992NjYwMnJCfHx8fjrr79KM9QCFScXIQRGjRqFiRMnwtfX1xBhFkpxj8vUqVMRFhaGffv24X//
+x8WLlyI9957r7TDfaHi5HLt2jUAwIcffohZs2Zh69atqFChAjp06IBHjx6VeswvUhLv/
9DQUHTr1g1Vq1YtjRALrTi5SCQS7NmzBydPnoSdnR2USiUWL16MiIgIVKhQwRBh56k4uXTq1An37t3D559/
DrVajcePH2PGjBkAgLt375Z6zDlK6zvy3r17eW4z5zFjCA0NxfDhwwEA3bt3R3JyMv7+
+2+jxFJYLHhLwaeffopffvkFFy9e1Gu/
ePEiWrdurdfWunVrXL58GRqNRtfWpEkT3f8lEgnc3d2RmJgIADh9+jSuXLkCOzs72NrawtbWFhUrVkRmZia
uXr1ailkVbOLEibqYbG1tjRpLSSgonzt37qB79+4YNGgQxo8fb4QIC+9FuUyfPh0nT57Erl27IJPJMHLkSJ
P+Sz2vXL799lukpqYiJCTEyNEVTX7HJTg4GB06dECTJk0wceJEfPnll/
j222+hUqmMGO2L5ZWLVqsFAMycORMDBgyAj48PVq5cCYlEgnXr1hkz3AIV9P6/
ffs2du7cibFjxxohuqLJKxchBCZPngxXV1ccOHAA0dHRCAwMRJ8+fQxaJBZVXrk0bNgQv/
zyC7788ktYW1vD3d0dNWrUgJubm95ZX2Mzl+/
IuLg4REdHY+jQoQAACwsLDB48GKGhoUaO7MUsjB2AOWrXrh26deuGkJAQjBo1qsjrP/
9zskQi0X1xpKWlwcfHB2vWrMm1nouLS7HiLSxnZ2fIZDIkJCTotSckJMDd3R3z58/Hu++
+W6oxlKSXyefff/
9Fx44dERAQgGXLlhki3Bd6mVycnZ3h7OyMunXron79+qhWrRqOHj2KVq1aGSL0POMpai579+7FkSNHcs3f7
uvri9dffx2//PJLqcedl5J6z/j7+yM7Oxs3btyAl5dXaYX7QsXJpVKlSgCABg0a6NoUCgVq1qyJ+Pj40g/
6BV722KxcuRJOTk7o27dvaYdaoOK+Z7Zu3YrHjx/
D3t4eAPDDDz9g9+7d+OWXX3RnSA2tuMdl2LBhGDZsGBISEmBjYwOJRILFixejZs2ahgq91L4j3d3dc430kL
MPd3f34gdcTKGhocjOzkblypV1bUIIKBQKfPfddwaPp7BM508fM/
PJJ59gy5Ytev126tevj0OHDuktd+jQIdStWxcymaxQ223evDkuX74MV1dX1K5dW+/
m4OBQojk8Ty6Xw8fHB5GRkbo2rVaLyMhItGrVKldMpq64+dy5cwcdOnTQna0yhTMIJXVscv6wMuaZxOLk8s
033+D06dM4deoUTp06he3btwN42gVowYIFRskDKLnjcurUKUilUri6uhoi7DwVJxcfHx8oFArdNQjA0+sQb
ty4AQ8PD4Pn8KyXOTZCCKxcuRIjR440an/
3HMXJJSMjAwByfX5JpVLd54AxvOx7xs3NDba2tggPD4dSqUTXrl3LTOz5adWqFc6ePav7pRcAdu/
eDXt7e70/Jg0hOzsbv/
76K7788kvd5+2pU6dw+vRpVK5cGWvXrjVoPEVizCvmzElQUJDo16+fXtuIESOEUqnUjdIQExMjpFKpmD9/
voiLixOrVq0SVlZWYuXKlbp18ro6u2nTpmLu3LlCCCHS09NFnTp1RIcOHcT+/
fvFtWvXxL59+8Sbb74pbt26VYoZPhUWFiYUCoVYtWqVuHDhgpgwYYJwdHTUu3r0eTdv3hQnT54U8+bNE7a2
tuLkyZPi5MmTIjU1tdTjLUhR87l9+7aoXbu26Ny5s7h9+7be8ETGVtRcjh49Kr799ltx8uRJcePGDREZGSk
CAgJErVq1RGZmpoGj11ec19mzrl+/
bjKjNBQ1l8OHD4uvvvpKnDp1Sly9elX89ttvwsXFRYwcOdLAkedWnOMybdo0UaVKFbFz504RGxsrxo4dK1x
dXfWGbzSW4r7O9uzZIwCIixcvGijSghU1l/v37wsnJyfRv39/
cerUKREXFyfeffddYWlpKU6dOmXg6PUV57h8++23IiYmRsTFxYnvvvtOWFlZia+//
tqAUT9VGt+ROcOSvfLKK+LUqVMiIiJCuLi4GGVYsk2bNgm5XC6SkpJyPfbee+8JX19fkx2lgQVvCcmr4L1+
/bqQy+V5DktmaWkpqlevLj7//HO9dQoqeIUQ4u7du2LkyJHC2dlZKBQKUbNmTTF+/
HiRnJxc0mnl6dtvvxXVq1cXcrlc+Pn5iaNHj75w+aCgIAEg123fvn0GibcgRcln5cqVeeZiKn87FiWXM2fO
iI4dO4qKFSsKhUIhPD09xcSJE8Xt27cNGHH+ivo6e5YpFbxCFC2XmJgY4e/
vLxwcHIRSqRT169cXCxcuNPofITmKelzUarV45513hKurq7CzsxNdunQR586dM1C0BSvO62zo0KEiICDAAN
EVTVFzOX78uHjllVdExYoVhZ2dnWjZsqXYvn27gaJ9saLmMmLECFGxYkUhl8tFkyZNxK+//mqgSHMrje/
IGzduiB49eggrKyvh7Ows3nnnHZGVlVXKmeTWu3dv0bNnzzwfO3bsmAAgvv76a5MseCVCmPDVKUREREREL8
n4nQ+JiIiIiEoRC14iIiIiMmsseImIiIjIrLHgJSIiIiKzxoKXiIiIiMwaC14iIiIiMmsseImIiIjIrLHgJ
SIiIiKzxoKXiMoNT09PLFmypES3OWrUKAQGBr5wmQ4dOuCtt94q0f2aisLkXxw3btyARCLBqVOn8l0mKioK
EokESUlJJbLPkt4eEZkOFrxEZHIkEskLbx9++GGxtnv8+HFMmDChZIMlIiKTZ2HsAIiInnf37l3d/
8PDwzFnzhzExcXp2mxtbXX/
F0JAo9HAwqLgjzMXF5eSDZReWlGOHxFRcfEMLxGZHHd3d93NwcEBEolEdz82NhZ2dnbYsWMHfHx8oFAocPD
gQVy9ehX9+vWDm5sbbG1t0aJFC+zZs0dvu893aZBIJFi+fDleffVVWFtbo06dOti8ebPucY1Gg7Fjx6JGjR
qwsrKCl5cXvv766zxjnjdvHlxcXGBv/3/t3X1M1dUfwPE3D8Xzg0LLOx/
ABZiYIEwioHADKVKZMRQkSghcOcIxAyRJHowaNxJo9GDWUqYrDdKcE5lDFqxInWOITG7MTKUpDiYiXURI7v
n90fzOK2D86vdg7PP675zP+X7OOd/vPx++91yuM+vXr2dkZGTC/
Q0PD5Odnc3MmTNxcHAgODiYxsbG+96T/v5+1q1bp80RERFBW1ubFi8qKmLRokXs2bMHT09PXFxcWLNmDb/
99ps2xmQyUVpaipeXFzY2NsyZM4d3331Xi7e3txMREYGdnR1ubm68+uqrGI1Gs/
vxxhtv4OrqipubG5s2bUIpZbZOk8lESUmJds/8/f355ptvtPidYwP3Pr+J/
PTTT4SGhmJra8sTTzxBU1PTfe/T/
v37WbBgATY2Nnh6elJWVmYWHx4eJjc3l9mzZ2NjY4OXlxdffPHFuLlu3rzJ888/T1hYGP39/
YyMjJCRkYFOp8PW1hYPDw9KSkruux4hxINBCl4hxD/Sm2+
+iV6vx2Aw4Ofnh9FoZNmyZTQ0NNDa2kp0dDQxMTF0dXXdN8/WrVuJj4/nzJkzLFu2jKSkJPr6+oA/
irdZs2ZRU1NDR0cHBQUF5OXlUV1dbZajoaEBg8FAY2Mje/fu5cCBA2zdunXCOTMyMjh+/
Dj79u3jzJkzrF69mujoaM6dOzfhNatXr6anp4e6ujpaWloIDAwkMjJSWyvA+fPnOXjwIIcPH+bw4cM0NTWh
1+u1+ObNm9Hr9eTn59PR0cFXX33Fo48+CsDg4CDPPfcc06ZN49SpU9TU1HDs2DEyMjK068vKyqiqqmLnzp3
88MMP9PX18e2335qts6SkhN27d/
Ppp59y9uxZNm7cyEsvvTSmUL33+U0kJyeHrKwsWltbCQkJISYmhmvXro07tqWlhfj4eNasWUN7eztFRUXk5
+dTVVWljVm7di179+6lsrISg8HAjh07zD4xuKO/v5+oqChMJhP19fW4urpSWVnJoUOHqK6uprOzky+//
BJPT88J1y6EeIAoIYR4gO3atUu5uLho7e++
+04B6uDBg3967YIFC9SHH36otT08PFRFRYXWBtSWLVu0ttFoVICqq6ubMOfrr7+u4uLitHZycrKaPn26Ghw
c1Pq2b9+uHB0d1ejoqFJKqSVLlqjMzEyllFKXLl1SVlZW6vLly2Z5IyMj1ebNm8ed8/
vvv1fOzs7q1q1bZv2PPfaY2rFjh1JKqcLCQmVvb68GBga0eE5OjgoODlZKKTUwMKBsbGzU559/
Pu4cn332mZo2bZoyGo1aX21trbK0tFRXr15VSiml0+lUaWmpFv/999/
VrFmz1MqVK5VSSt26dUvZ29urH3/80Sx3WlqaSkxMVEpN/vlduHBBAUqv14+Z77333jPLdf36daWUUi++
+KKKiooyy5OTk6N8fX2VUkp1dnYqQNXX14875518BoNB+fn5qbi4ODU8PKzFN2zYoCIiIpTJZLrv2oUQDx4
5NCWE+EdavHixWdtoNFJUVERtbS3d3d3cvn2boaGhP33De/
fbRQcHB5ydnenp6dH6Pv74Y3bu3ElXVxdDQ0OMjIywaNEisxz+/v7Y29tr7ZCQEIxGI7/+
+iseHh5mY9vb2xkdHcXHx8esf3h4GDc3t3HX2NbWhtFoHBMfGhri/
PnzWtvT0xMnJyetrdPptL0YDAaGh4eJjIwcdw6DwYC/
vz8ODg5aX1hYGCaTic7OTmxtbenu7iY4OFiLW1tbs3jxYu1Yw88//8zNmzeJiooyyz0yMkJAQIBZ373PbyI
hISFj5jMYDBPuYeXKlWZ9YWFhfPDBB4yOjnL69GmsrKxYsmTJfeeMioriySef5Ouvv8bKykrrT0lJISoqin
nz5hEdHc2KFSt49tlnJ7UPIcT/lxS8Qoh/
pLsLM4Ds7Gzq6+vZtm0bXl5e2NnZsWrVqvuepQV46KGHzNoWFhaYTCYA9u3bR3Z2NmVlZYSEhODk5MT777/
PyZMn//
K6jUYjVlZWtLS0mBVTwLgfrd+5RqfTjXvO19XVdVJ7sbOz+8trnqw7531ra2uZOXOmWczGxsasfe/
z+1+Y7D1Yvnw5+/
fvp6Ojg4ULF2r9gYGBXLhwgbq6Oo4dO0Z8fDxLly41O6MshHgwScErhJgSmpubSUlJITY2Fvij+Lp48eLfz
hkaGkp6errWd/
cb1Tva2toYGhrSCqoTJ07g6OjI7Nmzx4wNCAhgdHSUnp4ennnmmUmtIzAwkKtXr2Jtbf2Xz4x6e3tjZ2dHQ
0MD69atGxOfP38+VVVVDA4OasVoc3MzlpaWzJs3DxcXF3Q6HSdPniQ8PByA27dva+eJAXx9fbGxsaGrq+tP
36JO1okTJ8bMd/
e54nv30NzcbNbX3NyMj48PVlZWLFy4EJPJRFNTE0uXLp1wTr1ej6OjI5GRkTQ2NuLr66vFnJ2dSUhIICEhg
VWrVhEdHU1fXx/Tp0//
D+xWCPHfIgWvEGJK8Pb25sCBA8TExGBhYUF+fr72dvPv5Ny9ezdHjx5l7ty57Nmzh1OnTjF37lyzcSMjI6S
lpbFlyxYuXrxIYWEhGRkZWFqO/V6wj48PSUlJrF27lrKyMgICAujt7aWhoQE/
Pz+WL18+5pqlS5cSEhLCCy+8QGlpKT4+Ply5coXa2lpiY2MndTzA1taW3NxcNm3axMM
PP0xYWBi9vb2cPXuWtLQ0kpKSKCwsJDk5maKiInp7e9mwYQMvv/
yy9sW2zMxM9Ho93t7ePP7445SXl5v9SIOTkxPZ2dls3LgRk8nE008/zY0bN2hubsbZ2Znk5OR/
8wn8caTE29ub+fPnU1FRwfXr10lNTR13bFZWFkFBQRQXF5OQkMDx48f56KOP+OSTT4A/
jnwkJyeTmppKZWUl/
v7+XLp0iZ6eHuLj481ybdu2jdHRUSIiImhsbNT2q9PpCAgIwNLSkpqaGmbMmGH2ll0I8WCSglcIMSWUl5eT
mppKaGgo7u7u5ObmMjAw8Ldyvvbaa7S2tpKQkICFhQWJiYmkp6dTV1dnNi4yMhJvb2/
Cw8MZHh4mMTHxvj+OsWvXLt555x2ysrK4fPky7u7uPPXUU6xYsWLc8RYWFhw5coS33nqLV155hd7eXmbMmE
F4eLhWjE5Gfn4+1tbWFBQUcOXKFXQ6HevXrwfA3t6eo0ePkpmZSVBQEPb29sTFxVFeXq5dn5WVRXd3N8nJy
VhaWpKamkpsbCw3btzQxhQXF/PII49QUlLCL7/8gqurK4GBgeTl5U16nXfT6/
Xo9XpOnz6Nl5cXhw4dwt3dfdyxgYGBVFdXU1BQQHFxMTqdjrfffpuUlBRtzPbt28nLyyM9PZ1r164xZ86cC
ddWUVFhVvQ6OTlRWlrKuXPnsLKyIigoiCNHjoz7h40Q4sFiodQ9/0RRCCGEEEKIKUT+LBVCCCGEEFOaFLxC
CCGEEGJKk4JXCCGEEEJMaVLwCiGEEEKIKU0KXiGEEEIIMaVJwSuEEEIIIaY0KXiFEEIIIcSUJgWvEEIIIYS
Y0qTgFUIIIYQQU5oUvEIIIYQQYkqTglcIIYQQQkxp/wL7Z6Nzq42pRwAAAABJRU5ErkJggg==",
"text/plain": [
"<Figure size 800x400 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# import matplotlib.pyplot as plt\n",
"# import numpy as np\n",
"\n",
"# # Create Figure\n",
"# plt.figure(figsize=(8,4))\n",
"\n",
"# # Prepare Data\n",
"# x = [f\"0-{index}\" for index in range(12)]\n",
"# x[0] = \"None\"\n",
"# x[-1] = \"All\"\n",
"# y = [\n",
"# 0.8541862652869239,\n",
"# 0.8525519848771267,\n",
"# 0.8514664143803217,\n",
"# 0.8506616257088847,\n",
"# 0.8398104265402844,\n",
"# 0.8391345249294448,\n",
"# 0.8377358490566037,\n",
"# 0.8433962264150944,\n",
"# 0.8258801141769743,\n",
"# 0.816247582205029,\n",
"# 0.7917485265225934,\n",
"# 0.7019400352733686\n",
"# ][::-1]\n",
"\n",
"# # Stylize Figure\n",
"# plt.grid(color='#ECEFF1')\n",
"# plt.axvline(x=4, color=\"#EC407A\", linestyle=\"--\")\n",
"# plt.title(\"Effect of Frozen Encoder Blocks on Training Performance\")\n",
"# plt.ylabel(\"F1-score\")\n",
"# plt.xlabel(\"Trainable encoder blocks\")\n",
"\n",
"# # Plot Data\n",
"# plt.plot(x, y, color=\"black\")\n",
"\n",
"# # Additional Annotation\n",
"# plt.annotate(\n",
"# 'Performance stabilizing',\n",
"# xy=(4, y[4]),\n",
"# xytext=(4.5, y[4]-.05),\n",
 "# arrowprops=dict(\n",
"# arrowstyle=\"-|>\",\n",
"# connectionstyle=\"arc3\",\n",
"# color=\"#00ACC1\")\n",
"# )\n",
"# plt.savefig(\"multiple_frozen_blocks.png\", dpi=300, bbox_inches='tight')"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "sf785lzMjwiy"
},
"source": [
"## Few-shot Classification"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8ybeQ3j6kOk4"
},
"outputs": [],
"source": [
"from setfit import sample_dataset\n",
"\n",
"# We simulate a few-shot setting by sampling 16 examples per class\n",
"sampled_train_data = sample_dataset(tomatoes[\"train\"], num_samples=16)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3418,
"status": "ok",
"timestamp": 1719390247822,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "1Y55TDrmSqHm",
"outputId": "84ec7a9c-92ed-4277-dfff-f0cb4cc918d6"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283:
DeprecationWarning: `should_run_async` will not call `transform_cell` automatically
in the future. Please pass the result to `transformed_cell` argument and any
exception that happen during thetransform in `preprocessing_exc_tuple` in IPython
7.17 and above.\n",
" and should_run_async(code)\n",

"/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132:
FutureWarning: `resume_download` is deprecated and will be removed in version
1.0.0. Downloads always resume when possible. If you want to force a new download,
use `force_download=True`.\n",
" warnings.warn(\n",
"model_head.pkl not found on HuggingFace Hub, initialising classification
head with random weights. You should TRAIN this model on a downstream task to use
it for predictions and inference.\n"
]
}
],
"source": [
"from setfit import SetFitModel\n",
"\n",
"# Load a pre-trained SentenceTransformer model\n",
"model = SetFitModel.from_pretrained(\"sentence-transformers/all-mpnet-base-
v2\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 49,
"referenced_widgets": [
"630de7830edb4104938bef1c304129c0",
"736c3004e2ec48b2b3a8974e465eb838",
"54c6443169d84a3c9983d8c5b125c3e7",
"bbee47201aac4b8f96aa772cc751d1d8",
"9ad6e36197894003809d3ae6fa1b6a2c",
"4708f67e67194377a94ec08aa07fe688",
"c03a1bf3d2214c3680ab0b983dd582ef",
"e777041b66fa484090714094d9516f35",
"c5d50176a0a049769731d1c6fdb37f0f",
"aa6d83c964d741178534cd0fae6ccec9",
"318f91cb3cdd4f6e88681121de392207"
]
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719390248689,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "zZ10SpAXdNvC",
"outputId": "f9930b86-e077-4233-f7d7-16c5ec1e640d"
},
"outputs": [
{
 "data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "630de7830edb4104938bef1c304129c0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/32 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from setfit import TrainingArguments as SetFitTrainingArguments\n",
"from setfit import Trainer as SetFitTrainer\n",
"\n",
"# Define training arguments\n",
"args = SetFitTrainingArguments(\n",
" num_epochs=3, # The number of epochs to use for contrastive learning\n",
" num_iterations=20 # The number of text pairs to generate\n",
")\n",
"args.eval_strategy = args.evaluation_strategy\n",
"\n",
"# Create trainer\n",
"trainer = SetFitTrainer(\n",
" model=model,\n",
" args=args,\n",
" train_dataset=sampled_train_data,\n",
" eval_dataset=test_data,\n",
" metric=\"f1\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8HDP5-9wRYZS"
},
"outputs": [],
"source": [
"# from setfit import SetFitTrainer\n",
"\n",
"# # Create trainer\n",
"# trainer = SetFitTrainer(\n",
"# model=model,\n",
"# train_dataset=sampled_train_data,\n",
"# eval_dataset=test_data,\n",
"# metric=\"f1\",\n",
"# num_epochs=3, # The number of epochs to use for contrastive learning\n",
"# )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
 "colab": {
"base_uri": "https://localhost:8080/",
"height": 162
},
"executionInfo": {
"elapsed": 37840,
"status": "ok",
"timestamp": 1719390288851,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "3yaXeQJadWIS",
"outputId": "3bb59731-1360-4f4f-eaae-989a8547ca7c"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"***** Running training *****\n",
" Num unique pairs = 1280\n",
" Batch size = 16\n",
" Num epochs = 3\n",
" Total optimization steps = 240\n"
]
},
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='240' max='240' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [240/240 00:37, Epoch 3/0]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
 "# Training loop\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2455,
"status": "ok",
"timestamp": 1719390291303,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "PyRxiY32R3Jd",
"outputId": "51d00a55-8086-4456-b3c2-d8720342bcc8"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"***** Running evaluation *****\n"
]
},
{
"data": {
"text/plain": [
"{'f1': 0.8363988383349468}"
]
},
"execution_count": 75,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Evaluate the model on our test data\n",
"trainer.evaluate()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 129
},
"executionInfo": {
"elapsed": 10,
"status": "ok",
 "timestamp": 1719390291304,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "-aKIHJpCQdAm",
"outputId": "e2f786a4-78a4-4b92-d464-e3336d92edb2"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283:
DeprecationWarning: `should_run_async` will not call `transform_cell` automatically
in the future. Please pass the result to `transformed_cell` argument and any
exception that happen during thetransform in `preprocessing_exc_tuple` in IPython
7.17 and above.\n",
" and should_run_async(code)\n"
]
},
{
"data": {
"text/html": [
"<style>#sk-container-id-3 {color: black;background-color: white;}#sk-
container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-
color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor:
pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing:
border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-
arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-
container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-
container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before
{color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-
width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-
container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-
radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-
toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-
width: 100%;overflow: auto;}#sk-container-id-3 input.sk-
toggleable__control:checked~label.sk-toggleable__label-arrow:before
{content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-
toggleable__control:checked~label.sk-toggleable__label {background-color:
#d4ebff;}#sk-container-id-3 div.sk-label input.sk-
toggleable__control:checked~label.sk-toggleable__label {background-color:
#d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px
1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow:
hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-
estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted
black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-
container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-
3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid
gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label
{background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before
{content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-
box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial
{display: flex;flex-direction: column;align-items: center;background-color:
white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-
id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel
{display: flex;align-items: stretch;justify-content: center;background-color:
white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-
3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px
solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-
container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index:
1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-
item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3
div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-
container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3
div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-
sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-
3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-
block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align:
center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets
`[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !
important; }` so we also need the `!important` here to be able to override the
default hidden behavior on the sphinx rendered scikit-learn.org. See:
https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !
important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback
{display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-
container\"><div
class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter
environment, please rerun this cell to show the HTML representation or trust the
notebook. <br />On GitHub, the HTML representation is unable to render, please try
loading this page with nbviewer.org.</b></div><div class=\"sk-container\"
hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input
class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\"
type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-
toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div
class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></
div></div></div>"
],
"text/plain": [
"LogisticRegression()"
]
},
"execution_count": 76,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"model.model_head"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "L7NbUeSn-QSe"
},
"source": [
"## MLM"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
 "base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 1601,
"status": "ok",
"timestamp": 1719387317479,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "Z35PD47AXXnv",
"outputId": "45667c3b-9421-4406-e6e1-a406d803b8fd"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at bert-base-cased were not used when
initializing BertForMaskedLM: ['bert.pooler.dense.bias',
'bert.pooler.dense.weight', 'cls.seq_relationship.bias',
'cls.seq_relationship.weight']\n",
"- This IS expected if you are initializing BertForMaskedLM from the
checkpoint of a model trained on another task or with another architecture (e.g.
initializing a BertForSequenceClassification model from a BertForPreTraining
model).\n",
"- This IS NOT expected if you are initializing BertForMaskedLM from the
checkpoint of a model that you expect to be exactly identical (initializing a
BertForSequenceClassification model from a BertForSequenceClassification model).\n"
]
}
],
"source": [
"from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
"\n",
"# Load model for Masked Language Modeling (MLM)\n",
"model = AutoModelForMaskedLM.from_pretrained(\"bert-base-cased\")\n",
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 81,
"referenced_widgets": [
"5333a40e5cbc4bb5b89628ecaa6608b2",
"c7a67733b49e48df88a34329bd090349",
"40f95e431f264d3eb034f09f3f2e9e09",
"827d0cdd85c14d13b78da5c0d7f054c7",
"e8403bc0d63b4ec0aa679505a962dc34",
"bb38873892e847c6b646c9e3cc4c753d",
"496cfba4c0d04e399c6fde17772fa62c",
"e21c06a4d4ac4f5987b8e4f34c4b9ac8",
"c39ac7f5c60e4bf98e64174f21959d9a",
 "1caf892f43c94d50bb31f12b830462c6",
"526eb0a0a0d246578aa2fb726c7385e4",
"83160c120ff64c42af6d38bb3fb932b5",
"aa3c27c5a1254b19a7cfd04f28aee5d0",
"c350f4d70d5646fc973aa9e68711f830",
"1ca86de4b7174993942d43b0222f698f",
"cac994c5cc274c14ba2163c73fa4ef8c",
"4a1c759888cf472aab34c96be48e95b1",
"76aeb09a52f04f97aa006927b60bad84",
"60f082d7f4234bfa985c9241a7e1e9c0",
"8d84f304aca74bd99d7d624b29630495",
"71ea5effe952447fa40b578e64e1b868",
"933e2bdff863421abba4ba875a817dc0"
]
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1719387318933,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "zgLardIvEFTG",
"outputId": "f40b1e95-9a82-437e-ffd8-5e0fdb4be138"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "5333a40e5cbc4bb5b89628ecaa6608b2",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/8530 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "83160c120ff64c42af6d38bb3fb932b5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/1066 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
 "def preprocess_function(examples):\n",
" return tokenizer(examples[\"text\"], truncation=True)\n",
"\n",
"# Tokenize data\n",
"tokenized_train = train_data.map(preprocess_function, batched=True)\n",
"tokenized_train = tokenized_train.remove_columns(\"label\")\n",
"tokenized_test = test_data.map(preprocess_function, batched=True)\n",
"tokenized_test = tokenized_test.remove_columns(\"label\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "x7Yfg_2TECs_"
},
"outputs": [],
"source": [
"from transformers import DataCollatorForLanguageModeling\n",
"\n",
"# Masking Tokens\n",
"data_collator = DataCollatorForLanguageModeling(\n",
" tokenizer=tokenizer,\n",
" mlm=True,\n",
" mlm_probability=0.15\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wqnA-ROy7ZRG"
},
"outputs": [],
"source": [
"# from transformers import DataCollatorForWholeWordMask\n",
"\n",
"# # Masking Whole Words\n",
"# data_collator = DataCollatorForWholeWordMask(\n",
"# tokenizer=tokenizer,\n",
"# mlm=True,\n",
"# mlm_probability=0.15\n",
"# )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "OduApRY03iOE"
},
"outputs": [],
"source": [
"# Training arguments for parameter tuning\n",
"training_args = TrainingArguments(\n",
" \"model\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=16,\n",
 " per_device_eval_batch_size=16,\n",
" num_train_epochs=10,\n",
" weight_decay=0.01,\n",
" save_strategy=\"epoch\",\n",
" report_to=\"none\"\n",
")\n",
"\n",
"\n",
"# Initialize Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized_train,\n",
" eval_dataset=tokenized_test,\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 389
},
"executionInfo": {
"elapsed": 731739,
"status": "ok",
"timestamp": 1719388053472,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "v-cRB3QmEiXl",
"outputId": "3945078d-51b8-4ba9-9bd4-937cb4e7b858"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='5340' max='5340' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [5340/5340 12:10, Epoch 10/10]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
 " <tr>\n",
" <td>500</td>\n",
" <td>2.601700</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1000</td>\n",
" <td>2.377500</td>\n",
" </tr>\n",
" <tr>\n",
" <td>1500</td>\n",
" <td>2.313100</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2000</td>\n",
" <td>2.187500</td>\n",
" </tr>\n",
" <tr>\n",
" <td>2500</td>\n",
" <td>2.150400</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3000</td>\n",
" <td>2.096100</td>\n",
" </tr>\n",
" <tr>\n",
" <td>3500</td>\n",
" <td>2.059500</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4000</td>\n",
" <td>1.990300</td>\n",
" </tr>\n",
" <tr>\n",
" <td>4500</td>\n",
" <td>1.986100</td>\n",
" </tr>\n",
" <tr>\n",
" <td>5000</td>\n",
" <td>1.958500</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Save pre-trained tokenizer\n",
"tokenizer.save_pretrained(\"mlm\")\n",
"\n",
"# Train model\n",
"trainer.train()\n",
"\n",
"# Save updated model\n",
 "model.save_pretrained(\"mlm\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 1511,
"status": "ok",
"timestamp": 1719388054975,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "HfxN1p8TOg2v",
"outputId": "e9bcca59-cebb-4297-e675-b3d2a555d36e"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py:283:
DeprecationWarning: `should_run_async` will not call `transform_cell` automatically
in the future. Please pass the result to `transformed_cell` argument and any
exception that happen during thetransform in `preprocessing_exc_tuple` in IPython
7.17 and above.\n",
" and should_run_async(code)\n",
"Some weights of the model checkpoint at bert-base-cased were not used when
initializing BertForMaskedLM: ['bert.pooler.dense.bias',
'bert.pooler.dense.weight', 'cls.seq_relationship.bias',
'cls.seq_relationship.weight']\n",
"- This IS expected if you are initializing BertForMaskedLM from the
checkpoint of a model trained on another task or with another architecture (e.g.
initializing a BertForSequenceClassification model from a BertForPreTraining
model).\n",
"- This IS NOT expected if you are initializing BertForMaskedLM from the
checkpoint of a model that you expect to be exactly identical (initializing a
BertForSequenceClassification model from a BertForSequenceClassification model).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
">>> What a horrible idea!\n",
">>> What a horrible dream!\n",
">>> What a horrible thing!\n",
">>> What a horrible day!\n",
">>> What a horrible thought!\n"
]
}
],
 "source": [
"from transformers import pipeline\n",
"\n",
"# Load and create predictions\n",
"mask_filler = pipeline(\"fill-mask\", model=\"bert-base-cased\")\n",
"preds = mask_filler(\"What a horrible [MASK]!\")\n",
"\n",
"# Print results\n",
"for pred in preds:\n",
" print(f\">>> {pred['sequence']}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1719388054975,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "ogk1hJ4zOlAU",
"outputId": "316b16d0-065b-41e8-c35c-3ce3be481469"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
">>> What a horrible movie!\n",
">>> What a horrible film!\n",
">>> What a horrible mess!\n",
">>> What a horrible comedy!\n",
">>> What a horrible story!\n"
]
}
],
"source": [
"# Load and create predictions\n",
"mask_filler = pipeline(\"fill-mask\", model=\"mlm\")\n",
"preds = mask_filler(\"What a horrible [MASK]!\")\n",
"\n",
"# Print results\n",
"for pred in preds:\n",
" print(f\">>> {pred['sequence']}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "1sDqoG2NyeJO"
 },
"source": [
"## Named Entity Recognition\n",
"\n",
"Here are a number of interesting datasets you can also explore for NER:\n",
"* tner/mit_movie_trivia\n",
"* tner/mit_restaurant\n",
"* wnut_17\n",
"* conll2003"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KGDvXU-ZzJ3J"
},
"outputs": [],
"source": [
"from transformers import AutoModelForTokenClassification, AutoTokenizer\n",
"from transformers import DataCollatorWithPadding\n",
"from transformers import TrainingArguments, Trainer\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 298,
"referenced_widgets": [
"e8567cb28aff4974acf098b8fc4184ea",
"a0960ca186e9491fbff044109a197da2",
"1a56e5d0d26e49ad9bd6a919515de403",
"2fe1e77e27014f62964c94b62b66f049",
"2df3bbada30f48068cf8bd414f40d895",
"4457b446302442e4bcab06fc10eb15ae",
"23db3d47ec284ebfa6da24d9a58d0881",
"a2795e523b5a436d9f94b059e7252597",
"027f9ea43b6647ab91915020a2e25771",
"15c3f91714a74762989e35950d6c12a5",
"19b10fadedbd48288026ff8b21148c8f",
"e4c076fd3de24dba84f4d566d4fca48a",
"79c5f352cf7f4bf7ace576348e3c55d6",
"690b323dd52e48b4b130e63d377b4831",
"dbb5c772fc674bedaee90064d8e4163b",
"4fe4a9b1182e4383b9ffa8b0b0c0f321",
"6d12abf106954c2f9008c8505ed6fce3",
"ff5ebeb9078d4f8b975a8aa59977fae3",
"befc22d4789e49eb81793e4ce1210f41",
"18c3194ae3b8497fb325a424dfb95901",
"848c37b4a0224b4e994896ab75d695e4",
"bc720c483d74483790aa184e927bd742",
"7c9beb946034455e8f4e68cf9d3ed451",
"18f06f815052464aa05c788246507c1b",
"b58c221705ed490f801fe1cea0a50b60",
"c51e7b34c1c049c99753132ddf06a8bf",
 "ca05b822e80640728b7b98d75fc0a569",
"e70e0557dffe4ac4883151f464d88fb3",
"caab4f8ab71343dc9d949a4b387ff97d",
"114e90e1dc0b49149e94d9352a389352",
"95784022cda24c588e9986f5267e3922",
"1309c3d2eb8d4bd7b8aa8d7176b225d3",
"67d416d3edb747598eefad5de3969330",
"ec5ff2eceeaf425bb25a1580b1c86f35",
"322b02e7719b4795a0afcbbe99074678",
"376dbc09d30742cf9ea209d51b10bb46",
"453b539e6093481dabbd4bacc7b746e6",
"bb87ce6727e349df960b276b31477d42",
"ddf0589272894ee8bc63371aebae6c47",
"ce8a5082960345b28883dc1128550675",
"69a2c842c8e04a00bccfd9534ead0a3d",
"e4b0a87f816c4762b01e6952a4fb91f3",
"4956e10177f74c30a4dcec1aee5d84d1",
"930cc9ef722e4ad09fc5aec2bdb58f51",
"d133524c37da40a688ae41d0be4fb608",
"5068f7a72b444b368e3064b523d37316",
"20fc8f2be16b48a3a330cb07558c4a7d",
"b6871ea8d9ee464ca3d68d62c0264f65",
"2ba02b04fbee4db093abeffdc880ce1b",
"f4bce89187a74fda9f6b8412a886be56",
"2118f618461a4f45b864b8ca51561ecd",
"ca9ae83245b3466d961f8741b5b5ca22",
"c9aa5057f4c64210a50738ab0ae8d92f",
"bbc7a519dee04caca31b77c03b599cc1",
"5f33680650354398bd53b210141050ad",
"ed3c3a380c3a46e9b6eabe8e0df7f7a1",
"a1cfaa8574e34145a95b170954a6af0f",
"c94da6e530da46b284bee469474890cf",
"9e0395ec942f4b42b7983147701806f8",
"4730ed0d0d9a43d0b8b6a46f3351a91e",
"6eaad3c842a14b5b86fa908a18d86f20",
"9437374ca4e44034a7852cc05a345c18",
"31d743d1493746cdae66dff187dee185",
"4a2604352c6946e5b6ce02da0e82f00c",
"243f45f71e3a4385ac447112e2778915",
"20c7a8103eb44b61b2f221cc5ab5527b"
]
},
"executionInfo": {
"elapsed": 19287,
"status": "ok",
"timestamp": 1719388740030,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "YOQlMioIGY33",
"outputId": "3f5e52d8-e736-411c-e468-d63b48828109"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
 "model_id": "e8567cb28aff4974acf098b8fc4184ea",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading builder script: 0%| | 0.00/9.57k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e4c076fd3de24dba84f4d566d4fca48a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading readme: 0%| | 0.00/12.3k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"The repository for conll2003 contains custom code which must be executed to
correctly load the dataset. You can inspect the repository content at
https://hf.co/datasets/conll2003.\n",
"You can avoid this prompt in future by passing the argument
`trust_remote_code=True`.\n",
"\n",
"Do you wish to run the custom code? [y/N] y\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7c9beb946034455e8f4e68cf9d3ed451",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Downloading data: 0%| | 0.00/983k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ec5ff2eceeaf425bb25a1580b1c86f35",
"version_major": 2,
"version_minor": 0
},
 "text/plain": [
"Generating train split: 0%| | 0/14041 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d133524c37da40a688ae41d0be4fb608",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating validation split: 0%| | 0/3250 [00:00<?, ?
examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ed3c3a380c3a46e9b6eabe8e0df7f7a1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Generating test split: 0%| | 0/3453 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# The CoNLL-2003 dataset for NER\n",
"dataset = load_dataset(\"conll2003\", trust_remote_code=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1719388740030,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
 "id": "hOc5J7YgIPl8",
"outputId": "648a5ac4-0e61-44fa-fea7-60d1af1853fa"
},
"outputs": [
{
"data": {
"text/plain": [
"{'id': '848',\n",
" 'tokens': ['Dean',\n",
" 'Palmer',\n",
" 'hit',\n",
" 'his',\n",
" '30th',\n",
" 'homer',\n",
" 'for',\n",
" 'the',\n",
" 'Rangers',\n",
" '.'],\n",
" 'pos_tags': [22, 22, 38, 29, 16, 21, 15, 12, 23, 7],\n",
" 'chunk_tags': [11, 12, 21, 11, 12, 12, 13, 11, 12, 0],\n",
" 'ner_tags': [1, 2, 0, 0, 0, 0, 0, 0, 3, 0]}"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"example = dataset[\"train\"][848]\n",
"example"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719388740030,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "hEPNSdADeAWD",
"outputId": "fdb397ea-8105-44b2-b016-c29b69f61f7a"
},
"outputs": [
{
"data": {
"text/plain": [
"{'O': 0,\n",
" 'B-PER': 1,\n",
 " 'I-PER': 2,\n",
" 'B-ORG': 3,\n",
" 'I-ORG': 4,\n",
" 'B-LOC': 5,\n",
" 'I-LOC': 6,\n",
" 'B-MISC': 7,\n",
" 'I-MISC': 8}"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"label2id = {\n",
" 'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4,\n",
" 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8\n",
"}\n",
"id2label = {index: label for label, index in label2id.items()}\n",
"label2id"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 886,
"status": "ok",
"timestamp": 1719388740915,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "OtRCaz15AyjC",
"outputId": "b22d94ad-b666-4f4c-a76b-07102752a88c"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Some weights of BertForTokenClassification were not initialized from the
model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias',
'classifier.weight']\n",
"You should probably TRAIN this model on a down-stream task to be able to use
it for predictions and inference.\n"
]
}
],
"source": [
"from transformers import AutoModelForTokenClassification\n",
"\n",
 "# Load tokenizer\n",
"tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n",
"\n",
"# Load model\n",
"model = AutoModelForTokenClassification.from_pretrained(\n",
" \"bert-base-cased\",\n",
" num_labels=len(id2label),\n",
" id2label=id2label,\n",
" label2id=label2id\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2,
"status": "ok",
"timestamp": 1719388740915,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "QOrlt03kIt_f",
"outputId": "4f58559f-5d3e-43ed-a2f9-e6a5d13f5ba0"
},
"outputs": [
{
"data": {
"text/plain": [
"['[CLS]',\n",
" 'Dean',\n",
" 'Palmer',\n",
" 'hit',\n",
" 'his',\n",
" '30th',\n",
" 'home',\n",
" '##r',\n",
" 'for',\n",
" 'the',\n",
" 'Rangers',\n",
" '.',\n",
" '[SEP]']"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Split individual tokens into sub-tokens\n",
"token_ids = tokenizer(example[\"tokens\"], is_split_into_words=True)
[\"input_ids\"]\n",
"sub_tokens = tokenizer.convert_ids_to_tokens(token_ids)\n",
"sub_tokens"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 113,
"referenced_widgets": [
"2374a01e18eb42af8f9e9448c7500f3f",
"052d9ca5cba84efda37ddcc4e7122dd3",
"a11d42f341fa451b85d1042c45041aee",
"b0dfa37334294bd387caf4193ddb4fa8",
"c6a90623259049c89167579c116b8a2b",
"079ab07a92cb4e2985d04dd8428b4286",
"ab1366b8f75c4d35a70b00a555d45fff",
"d44f9fd0544f4dca964aa7aa0e792f77",
"eaf47f4519cb4839a8b33c4c87f1eb2f",
"99ffc8e69feb461095323a52cd4153c5",
"c558a4dfcec34484899578274ca46d38",
"463e4b109eaf4868a6abf5c979f279ba",
"d4ccf5b35fab4accbce478de57fd52ef",
"dfa3d3c4f64e487eadc1c288cdbf0463",
"d997aee4803042c29b88025d9500ec12",
"8505564f6157421c99e675ab7065dc9b",
"b907032d76ea427385e59eb1f7c6a5a7",
"a79dd7dc444243dcbe3d056f9125857b",
"e1ec63869ec24a5eb4e2bef5c646695c",
"3816be92ddad4795b714158f1ec8f1db",
"4fa53ee583bd4a77a685b4b0ed01ea50",
"53790691595e41c29a84a4e02dbbcbfd",
"15b5d717e0824b2292b90b4d679e2ca9",
"02d997424a4b4e26b60023e938edb44e",
"def65a08a1264375aa007ce1d9205f3a",
"5ce6635fc9ed4406a63a87baf1e28f17",
"05679dbc7df549bd9788accb2f7a62d1",
"e648eec3fd584b6ca370d5c2d804b092",
"58a9e06b27d34500ab344ad0b77281a6",
"7f44960348fa48109a8dc0b9f6780f52",
"889fd835725e4f2480fc8c1dd329299e",
"8000c805502a4657bc2c657f63af75d0",
"1fef2fd92da448c8a509ed6fccf71ac7"
]
},
"executionInfo": {
"elapsed": 4419,
"status": "ok",
"timestamp": 1712752415862,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "7s5A3mzj6TXf",
 "outputId": "b92cc409-96bd-43a0-9e3b-955003157f36"
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2374a01e18eb42af8f9e9448c7500f3f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/14041 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "463e4b109eaf4868a6abf5c979f279ba",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/3250 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "15b5d717e0824b2292b90b4d679e2ca9",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Map: 0%| | 0/3453 [00:00<?, ? examples/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def align_labels(examples):\n",
" token_ids = tokenizer(examples[\"tokens\"], truncation=True,
is_split_into_words=True)\n",
" labels = examples[\"ner_tags\"]\n",
"\n",
" updated_labels = []\n",
" for index, label in enumerate(labels):\n",
"\n",
" # Map tokens to their respective word\n",
" word_ids = token_ids.word_ids(batch_index=index)\n",
" previous_word_idx = None\n",
" label_ids = []\n",
 " for word_idx in word_ids:\n",
"\n",
" # The start of a new word\n",
" if word_idx != previous_word_idx:\n",
"\n",
" previous_word_idx = word_idx\n",
" updated_label = -100 if word_idx is None else label[word_idx]\
n",
" label_ids.append(updated_label)\n",
"\n",
" # Special token is -100\n",
" elif word_idx is None:\n",
" label_ids.append(-100)\n",
"\n",
" # If the label is B-XXX we change it to I-XXX\n",
" else:\n",
" updated_label = label[word_idx]\n",
" if updated_label % 2 == 1:\n",
" updated_label += 1\n",
" label_ids.append(updated_label)\n",
"\n",
" updated_labels.append(label_ids)\n",
"\n",
" token_ids[\"labels\"] = updated_labels\n",
" return token_ids\n",
"\n",
"tokenized = dataset.map(align_labels, batched=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 3,
"status": "ok",
"timestamp": 1712752662929,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "gIWOMzE-AhTu",
"outputId": "66cfe4c4-406c-4362-d93f-7999078040f3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Original: [1, 2, 0, 0, 0, 0, 0, 0, 3, 0]\n",
"Updated: [-100, 1, 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, -100]\n"
]
}
],
 "source": [
"# Difference between original and updated labels\n",
"print(f\"Original: {example['ner_tags']}\")\n",
"print(f\"Updated: {tokenized['train'][848]['labels']}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "bA9nE7E6g97p"
},
"outputs": [],
"source": [
"import evaluate\n",
"\n",
"# Load sequential evaluation\n",
"seqeval = evaluate.load(\"seqeval\")\n",
"\n",
"def compute_metrics(eval_pred):\n",
" # Create predictions\n",
" logits, labels = eval_pred\n",
" predictions = np.argmax(logits, axis=2)\n",
"\n",
" true_predictions = []\n",
" true_labels = []\n",
"\n",
" # Document-level iteration\n",
" for prediction, label in zip(predictions, labels):\n",
"\n",
" # token-level iteration\n",
" for token_prediction, token_label in zip(prediction, label):\n",
"\n",
" # We ignore special tokens\n",
" if token_label != -100:\n",
" true_predictions.append([id2label[token_prediction]])\n",
" true_labels.append([id2label[token_label]])\n",
"\n",
" results = seqeval.compute(predictions=true_predictions,
references=true_labels)\n",
" return {\"f1\": results[\"overall_f1\"]}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "m0S4ZajdCaJl"
},
"outputs": [],
"source": [
"from transformers import DataCollatorForTokenClassification\n",
"\n",
"# Token-classification Data Collator\n",
"data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
]
},
{
"cell_type": "code",
 "execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"executionInfo": {
"elapsed": 170605,
"status": "ok",
"timestamp": 1712753138553,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "S76kRYZtBr5c",
"outputId": "8c2e866b-e9a9-49e3-ef08-56aab5aacdb4"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <div>\n",
" \n",
" <progress value='878' max='878' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [878/878 02:49, Epoch 1/1]\n",
" </div>\n",
" <table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: left;\">\n",
" <th>Step</th>\n",
" <th>Training Loss</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <td>500</td>\n",
" <td>0.047500</td>\n",
" </tr>\n",
" </tbody>\n",
"</table><p>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"TrainOutput(global_step=878, training_loss=0.04094860494001037,
metrics={'train_runtime': 169.4752, 'train_samples_per_second': 82.85,
'train_steps_per_second': 5.181, 'total_flos': 351240792638148.0, 'train_loss':
0.04094860494001037, 'epoch': 1.0})"
 ]
},
"execution_count": 54,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Training arguments for parameter tuning\n",
"training_args = TrainingArguments(\n",
" \"model\",\n",
" learning_rate=2e-5,\n",
" per_device_train_batch_size=16,\n",
" per_device_eval_batch_size=16,\n",
" num_train_epochs=1,\n",
" weight_decay=0.01,\n",
" save_strategy=\"epoch\",\n",
" report_to=\"none\"\n",
")\n",
"\n",
"# Initialize Trainer\n",
"trainer = Trainer(\n",
" model=model,\n",
" args=training_args,\n",
" train_dataset=tokenized[\"train\"],\n",
" eval_dataset=tokenized[\"test\"],\n",
" tokenizer=tokenizer,\n",
" data_collator=data_collator,\n",
" compute_metrics=compute_metrics,\n",
")\n",
"trainer.train()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 141
},
"executionInfo": {
"elapsed": 14521,
"status": "ok",
"timestamp": 1712753153056,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "ds5osPr9T0pq",
"outputId": "28e6836f-5eae-43c2-a37f-b92acb9a42b6"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
 " <div>\n",
" \n",
" <progress value='216' max='216' style='width:300px; height:20px;
vertical-align: middle;'></progress>\n",
" [216/216 00:09]\n",
" </div>\n",
" "
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"{'eval_loss': 0.16888542473316193,\n",
" 'eval_f1': 0.9180087380808113,\n",
" 'eval_runtime': 14.5731,\n",
" 'eval_samples_per_second': 236.943,\n",
" 'eval_steps_per_second': 14.822,\n",
" 'epoch': 1.0}"
]
},
"execution_count": 55,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Evaluate the model on our test data\n",
"trainer.evaluate()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"executionInfo": {
"elapsed": 2674,
"status": "ok",
"timestamp": 1712753204334,
"user": {
"displayName": "Maarten Grootendorst",
"userId": "11015108362723620659"
},
"user_tz": -120
},
"id": "Q0PAXyzT-N45",
"outputId": "d29825c0-2be2-48dd-c685-fad114afb369"
},
"outputs": [
{
"data": {
 "text/plain": [
"[{'entity': 'B-PER',\n",
" 'score': 0.99534035,\n",
" 'index': 4,\n",
" 'word': 'Ma',\n",
" 'start': 11,\n",
" 'end': 13},\n",
" {'entity': 'I-PER',\n",
" 'score': 0.9928328,\n",
" 'index': 5,\n",
" 'word': '##arte',\n",
" 'start': 13,\n",
" 'end': 17},\n",
" {'entity': 'I-PER',\n",
" 'score': 0.9954301,\n",
" 'index': 6,\n",
" 'word': '##n',\n",
" 'start': 17,\n",
" 'end': 18}]"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from transformers import pipeline\n",
"\n",
"# Save our fine-tuned model\n",
"trainer.save_model(\"ner_model\")\n",
"\n",
"# Run inference on the fine-tuned model\n",
"token_classifier = pipeline(\n",
" \"token-classification\",\n",
" model=\"ner_model\",\n",
")\n",
"token_classifier(\"My name is Maarten.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "MAfBhqVwC61e"
},
"outputs": [],
"source": []
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"authorship_tag": "ABX9TyOzKODTW4KPEpkvmaAMgqlD",
"gpuType": "T4",
"provenance": [],
"toc_visible": true
},
"kernelspec": {
 "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}