Jul 9, 2023 · We find that, although. GPT-3.5 is cheaper than human annotators, fine- tuning T5-XXL and then distilling a small model is more cost-efficient ...
May 2, 2023 · We show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a ...
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models. This repository contains resources for the ACL 2023 paper Distill or Annotate? Cost-Efficient ...
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models. Junmo Kang, Wei Xu, Alan Ritter. Junmo.kang@gatech.edu; {wei.xu, alan.ritter}@cc.gatech.edu.
People also ask
What is the difference between model distillation and fine-tuning?
How to efficiently fine tune your machine learning models?
May 2, 2023 · We find that, for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, a ...
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models. Page 2. Introduction. 2. Q. Given a fixed budget, how to build a compact model in a cost- ...
Jul 1, 2023 · Junmo Kang, Wei Xu, and Alan Ritter. Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models. Retrieved from https://par.nsf.gov/ ...
May 3, 2023 · In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through our extensive experiments on six ...
Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models. J. Kang, W. Xu, and A. Ritter. ACL (1), page 11100-11119. Association for Computational ...
Cost-Efficient Fine-Tuning of Compact Models, Junmo Kang, Wei Xu, Alan ... Option 1: Annotate, then fine-tune T5-Small. Option 2: Fine tune T5-XXL (11B) ...