Open-Core Computer Vision
This document outlines the sortability study conducted by CRAB Traceability Systems on
Colgate 75ml tubes in Luxembourg on November 21st. The study had two main objectives:
1. To create a real-life dataset of Colgate 75ml tubes within a sorting facility.
2. To develop a computer vision (CV) model capable of recognizing, tracking, and
counting these tubes in real-world conditions.
The resulting dataset, along with the trained CV model, provides valuable enablers for
companies involved in packaging recognition and sorting (e.g., Glacier, GreyParrot,
PolyPerception). These enabler are:
1. The collection of sortability data from various locations at scale, adopting an open-core
approach to understand how many Colgate tubes are being sorted in practice.
2. Enhanced performance for sorting machine providers, allowing for improved
identification and separation of Colgate tubes, thereby increasing their overall
sortability.
In doing so, Colgate takes the next step beyond redesigning its packaging, moving closer to
achieving real-world sortability and recycling outcomes.
Figure 1. Sample images from the data set and the model
The following chapters detail the study methodology, results, and the computer vision
development process.
1) Introduction
CRAB Traceability Systems (https://crab-ts.com/) is a spin-off from the University of
Luxembourg, with expertise in Design-for-Recycling, sorting and recycling process analysis,
computer vision and robotics, co-founded by Dr. Jeff Mangers, Dr. Alexej Simeth, and Dr. Atal
Kumar, with the support of their advisors Marc Jacobs and Bernard Merkx. Marc Jacobs is a
co-founder of LuxPet, now Plastipak’s Luxembourgish facility, while Bernard Merkx is the
former president of the European Association of Plastics Recyclers (PRE) and the current
managing director of the European Association of Plastics Converters (EUPC).
On November 8, CRAB received 1,000 freshly produced Colgate tubes. To simulate real-life
conditions, these tubes were prepared to look used and dirty to simulate their condition after
use by the consumer. On November 21, a team of four CRAB employees visited a
Luxembourg sorting plant to carry out the required sortability tests. Equipped with 500 Colgate
tubes and their mobile camera system, the team collected the required data.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 1 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
Following the field tests, CRAB spent two weeks processing the data, including annotating,
categorizing, and training the computer vision (CV) model. The results of the study were then
presented to Adrian Sen on December 9th during an online meeting.
The subsequent sections of this document detail the methodology, review the results, and
explore the potential use cases and value propositions for Colgate based on this study.
2) Methodology behind sorting test, data-set generation and model training
The methodology for this study is divided into several key steps, each contributing to the
overall process of data collection, analysis, and model development.
Below is a summary of the process:
1. Tubes Preparation
Freshly produced Colgate tubes were prepared to resemble used and dirty tubes,
simulating their post-consumer state.
2. Sorting Facility Analysis & Camera System Installation
A thorough analysis of the sorting facility was conducted to identify the optimal setup
for the mobile camera system.
3. Tubes Introduction & Data Capturing
The prepared tubes were introduced into the sorting system, and data—including
images and videos—was captured using the mobile camera system.
4. Data Retrieval & Preparation
The captured data was retrieved and checked for quality, including evaluating and
addressing issues like image blurriness.
5. Dataset Generation (Annotation & Labeling)
The captured images were annotated and labeled to generate a high-quality dataset
specific to Colgate 75ml tubes.
6. Model Training, Parametrization, & Benchmarking
A computer vision model for the Colgate 75ml tubes was trained, its parameters
optimized, and its performance benchmarked.
7. Report Preparation
The results were compiled into a comprehensive presentation and report.
Figure 2. Getting from production quality to end-of-use appearance
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 2 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
This entire process was completed within two weeks. However, with additional collaboration
and budgetary support from Colgate, we aim to streamline the process. Our target is to reduce
the timeframe to 1–2 days initially, and eventually to just 10 minutes, through a mobile AI
camera system.
It is also important to note that, thanks to our collaboration with the Luxembourgish
Supercomputer, all captured data and training activities were conducted and stored entirely in
Luxembourg. This ensures full control over the data while keeping it compliant with EU
regulations by remaining within the European Union.
3) Set-up & Results
This chapter provides a simplified overview of the sorting facility set-up, followed by an in-
depth discussion of the results achieved during the study.
Set-up
The figure below illustrates a simplified schematic of the sorting facility set-up, highlighting the
placement of the camera installations and the points where Colgate tubes were introduced
and retrieved.
First Round
➢ A first camera was installed before the NIR HDPE sorter, where the tubes were
introduced into the sorting process.
➢ A second camera was placed after the NIR HDPE sorter to capture data, as
well as retrieve tubes again manually.
➢ During this run, 500 Colgate tubes were introduced into the stream, and 367
tubes were manually retrieved. However, this number does not reflect the
efficiency of the NIR HDPE sorter due to limitations in manual retrieval.
➢ A future study with three cameras - one at the input and one at each output -
could provide a complete analysis of sorting efficiency for Colgate tubes at a
machine or facility-wide level.
Second Round
➢ The first camera was moved to just behind the drum to capture a more crowded
environment, where medium-sized packaging moves on a single conveyor belt.
➢ The 367 tubes retrieved from the first round were reintroduced here, and 229
tubes were subsequently retrieved manually after passing through the NIR
HDPE sorter.
Figure 3. Simplified overview of sorting facility with camera installations
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 3 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
Results
From the two rounds of testing, 1,448 unique images of Colgate tubes were collected. Notably,
no image contained the same tube twice (i.e., no consecutive frames were used). This dataset
served as the basis for training a computer vision model with the following performance
metrics:
➢ mAP50: 94.33%
➢ Precision: 94.05%
➢ Recall: 90.31%
These results are particularly impressive given the time constraints and the use of a mobile,
non-fixed camera system. The model’s performance is comparable to results typically
achieved by fixed camera systems, which usually promise metrics around 96–97%.
Screenshots
The figure below shows screenshots from the model’s recognition performance across various
locations:
➢ After the drum (second round)
➢ Before the NIR HDPE (first round)
➢ After the NIR HDPE (both rounds)
Figure 4. Screenshots from model recognition performance at different locations
Future Improvements
The current study highlights the potential of CRAB’s mobile camera system to achieve high
performance in waste recognition and sortability testing. However, a more extensive follow-up
study could provide:
➢ Detailed analysis of sorting efficiency with additional cameras (input and output
monitoring).
➢ Expansion of the dataset size and diversity to improve model robustness further.
➢ Advanced augmentation techniques to optimize the performance of the computer
vision model.
➢ More detailed information about the dataset & model parameters
In summary, this initial study demonstrates the feasibility and effectiveness of leveraging a
mobile camera system and computer vision for sortability analysis, paving the way for future
advancements and larger-scale evaluations.
These results not only demonstrate the potential of leveraging computer vision for packaging
sortability but also lay the groundwork for creating tangible value for Colgate by enabling direct
application and strategic sharing of the dataset and model across relevant stakeholders.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 4 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
4) Applications & Use-Cases
The goal of this project, as refined through discussions with Adrian Sen, is to establish an
Open-Core dataset and computer vision (CV) model tailored specifically for Colgate tubes.
This Open-Core approach enables secure sharing of CV datasets and models with key
stakeholders - such as sorting and camera system companies - while incorporating robust
security protocols.
These aspects have already been internally developed and reviewed with input from Prof.
Thomas Engel, an expert in network and system security (link).
This chapter explores the importance of this approach, the value it creates for Colgate, and
how it is becoming increasingly critical given regulatory and market trends.
What?
The project focuses on creating specific datasets for Colgate tubes and training corresponding
CV models under an Open-Core framework. This framework allows CRAB to:
1. Develop and maintain a centralized dataset and CV model for Colgate.
2. Share these assets in a secure and controlled environment where partners can either:
➢ Use the dataset to train models tailored to their needs.
➢ Directly access the pre-trained, parameterized model.
This system ensures oversight, accountability, and proper usage of shared assets while
enabling seamless reporting back to Colgate.
How?
The exchange is based on privacy-preserving federated machine learning, leveraging Secure
Multi-Party Computation Protocols (SMPC). In this setup:
➢ CRAB manages the creation and training of the Colgate-specific model.
➢ Other CV partners, operating within sorting and recycling facilities, can utilize the
dataset or model to improve their operations.
➢ These partners, in turn, provide reporting data back to Colgate, delivering actionable
insights on tube recognition and sortability.
This secure and collaborative approach facilitates data collection while maintaining control
and compliance within the EU regulatory framework.
Why?
The increasing importance of sortability and recyclability claims, driven by regulations like the
PPWR and Green Claims initiatives, makes accurate data and proof essential.
However:
➢ Collecting relevant data and training brand-specific models is complex and resource-
intensive.
➢ Existing CV providers may lack the initial dataset needed to recognize Colgate tubes
effectively.
By taking a proactive approach, Colgate can lead the way in providing pre-prepared datasets
and models to all relevant partners, reducing the effort required by individual providers and
enabling faster implementation. Colgate is thus taking the next step beyond the design phase
and moving closer to actual sortability and recyclability.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 5 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
Value for Colgate
The open core concept positions CRAB as an independent intermediary that ensures
Colgate's data is not tied to a single provider but is shared across the entire ecosystem. In
Europe alone, there are around 1,200 sorting plants where it is impossible for just one partner
to access and receive all the data over time. It is therefore only possible to obtain usable data
through cooperation and interaction between several partners, as proposed by CRAB.
This strategy benefits Colgate by:
1. Enabling collaboration with a wide array of partners, including GreyParrot, WasteAnt,
Glacier, ZenRobotics, RecycleEye, Waste Robotics, PolyPerception, WeSortAI and
others, ensuring maximum reach and compatibility.
2. Consolidating and standardizing reporting from multiple facilities into a single,
comprehensive report.
3. Promoting sustainability by reducing the need to repeatedly waste new tubes for
training purposes. A single model and dataset can be shared across all stakeholders.
Financial Model
The financial proposal includes two components:
1. Onboarding Fee: A one-time service fee for generating the dataset and training the
model.
2. Subscription Service: A recurring fee for ongoing access to bundled data and reports
from multiple locations and providers.
The subscription fee is shared among CRAB and participating partners, who contribute to data
collection and model deployment.
Figure 5. Computer Vision & Reporting Data Flow
CRAB acts as the bridge between Colgate’s production (beginning-of-life) and
sorting/recycling facilities (end-of-life), facilitating data exchange, model sharing, and
reporting.
The next part explains possible next steps, which are divided into short-term and middle/ long-
term options, although a combination would of course be ideal.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 6 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
5) Next Steps
The next steps are divided into short-term and long-term options to ensure a gradual, practical
approach that delivers immediate benefits while working towards larger, scalable solutions.
The aim is to move efficiently through each phase and achieve actionable milestones that
pave the way for further progress.
Short-Term Objectives
1. Enhance the Current Study
Expand the dataset, improve the model’s performance, and incorporate additional
product groups into the dataset and computer vision (CV) model for broader
applicability.
2. Data Collection in Luxembourg
Begin systematic data collection in Luxembourg, leveraging the country as a controlled
industrial laboratory. This allows us to generate reliable sortability reports from a single
region before scaling further. In cases where no CV provider is available in a specific
facility, CRAB can supply the required installations to ensure data collection continuity.
3. Engage with Industry Consortiums
Share the findings with tube-focused industry groups such as STINA. Extending the
study results to other Colgate partners or consortium members will foster broader
applicability, increased usage, and greater industry influence over time.
Middle- and Long-Term Objectives
4. Benchmarking Across Locations
Deploy CRAB’s mobile camera system to benchmark sortability in various facilities and
regions. For example, the system could travel across the U.S., collecting data to
assess the sortability of Colgate tubes in regions with high demand or limited recycling
infrastructure.
5. Expand Collaboration with CIRCPACK
Partner with CIRCPACK to scale dataset generation and model training for additional
product types, such as Palmolive, Sanex, and Ajax. This partnership could also yield
deeper insights into sortability behavior, leveraging CRAB’s mobile camera system to
pinpoint and address challenges.
This point has already been mentioned to Filipe Vieira De Castro during an online
meeting on 10.12.2024.
6. Engage CV Providers
Begin discussions with CV technology providers, starting with existing Colgate
partners like Glacier. Sharing the Open-Core database and trained models with Glacier
allows for continued development and practical testing. Once proven successful, the
initiative can be extended to other CV providers, maximizing the number of sorted and
recycled tubes globally.
Additional Opportunities
This roadmap is non-exhaustive and reflects the outcomes of discussions to date, alongside
additional value propositions identified by CRAB. All results and progress can be integrated
into CRAB’s web-based platform, which digitally replicates sorting and recycling facilities. This
platform also serves as a tool for Design-for-Recycling simulations and assessments, making
collected data actionable for future packaging designs and sustainability strategies.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 7 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
6) Conclusion
The presented results and concepts represent a foundational step in demonstrating how
Colgate can advance beyond design changes to ensure real sortability and recyclability. By
leveraging real-world data and rigorous computer vision (CV) methodologies, this study
establishes a clear path for Colgate to measure, improve, and validate the performance of its
packaging in sorting and recycling environments.
The development of the Colgate-specific dataset and CV model not only enables precise
identification, tracking, and counting of Colgate tubes but also introduces an innovative Open-
Core approach. This ensures the models and datasets are accessible to a broad network of
stakeholders across the recycling value chain, fostering transparency and collaboration.
These advancements create measurable value by providing the data needed to support
compliance with emerging regulations such as the PPWR and Green Claims, while actively
increasing the sortability and recyclability of Colgate products.
Short-term next steps, such as expanding the dataset, improving model performance, and
sharing findings with industry groups, lay the groundwork for immediate progress. Long-term
initiatives, including benchmarking global sortability, collaborating with CIRCPACK for multi-
product dataset generation, and engaging CV providers, align with Colgate’s strategic goal to
drive systemic improvement across the recycling ecosystem.
The honest evaluation and methodologies outlined in this report reaffirm CRAB’s commitment
to supporting Colgate on its journey to operationalize sustainable packaging practices. By
using real, trustworthy data, Colgate is uniquely positioned to not only meet regulatory
requirements but also set a benchmark for responsible, scalable, and innovative packaging
design and recycling practices within the industry.
This is just the beginning. With a step-by-step approach, strong partnerships and robust data
collection, Colgate can continue to lead the way in achieving recyclability and sustainability,
backed by transparent and actionable insights.
In Luxembourg, on 12.12.2024.
Dr. Jeff Mangers
Co-founder & CEO of CRAB Traceability Systems
This document is confidential and intended solely for the internal use of Colgate/ CRAB Traceability Systems. Unauthorized
distribution, copying, or sharing with third parties is strictly prohibited.
All rights reserved © CRAB Traceability Systems.
CRAB Traceability Systems Sàrl-S
University of Luxembourg Incubator https://crab-ts.com/
6A, avenue des Hauts-Fourneaux 8 jeff.mangers@crab-ts.com
L-4362 Esch-sur-Alzette/ Luxembourg © by CRAB Traceability Sytems - Confidential
