3. Development phase: Data pre-processing
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna.
Why should I do this?
Address quality issues and reconstruct the dataset from the initial raw data into a format the model can use, ensuring required criteria are met. This helps ensure that the model achieves the goals established in the problem definition phase.
- Workflow activities Miro Board
Target users
- CTOs and technical leaders
- Data scientists and ML engineers
- Product managers
- Research organizations, think tanks
Checklists in play
- Checklist 1: Risk Identification & Mitigation (Q7, Q8)
- Checklist 3: Legal & Regulatory Compliance (Q8)
Questions to consider
Checklist 1 · Q7
How were pre-processing activities assessed to ensure they do not introduce errors or distortions that could affect outcomes?
Checklist 1 · Q8
What evidence shows that all data pre-processing steps are documented in a transparent, reproducible, and version-controlled manner?
Checklist 3 · Q8
Can data subjects (i.e., individuals whose personal data is processed) meaningfully exercise their data protection rights, including:
- Right to be informed;
- Right to rectify personal information;
- Right to revoke consent for personal data processing;
- Right to restrict data processing;
- Any other rights afforded under the applicable data protection laws of the country wherein the data subject resides, or where the data processor (AI developer) is incorporated.
Investment types
Every investment project is unique
The application of the six steps will vary accordingly. To provide examples that align with your project, common characteristics of AgDev investments were researched and three ‘investment types’ were developed.
AgriConnect: a digital solutions investment
[MISSING – no content in source row assigned to this card]
Participatory co-creation, feedback, and model refinement
- What is the process followed in determining which datasets are included and excluded?
Lessons learned and challenges that arose
[MISSING – no content in source row assigned to this card]
Decisions taken and documentation maintained
The team excluded several datasets until consent and licensing status could be clarified. A data inclusion/exclusion log was created documenting justification for each dataset.
AgroThrive: a policy and advocacy investment
[MISSING – no content in source row assigned to this card]
Participatory co-creation, feedback, and model refinement
[MISSING – no content in source row assigned to this card]
Lessons learned and challenges that arose
[MISSING – no content in source row assigned to this card]
Decisions taken and documentation maintained
A dataset weighting framework was introduced to intentionally diversify source representation. Metadata tags were added identifying funding source, geography and stakeholder origin. Exclusion/inclusion decisions and rationale were documented in a dataset governance register.
NGBT: a field research investment
[MISSING – no content in source row assigned to this card]
Participatory co-creation, feedback, and model refinement
- Does preprocessing unintentionally simplify nuanced agricultural trade-offs?
Lessons learned and challenges that arose
[MISSING – no content in source row assigned to this card]
Decisions taken and documentation maintained
Dataset inclusion criteria and representation reviews were documented in a dataset governance register and bias assessment report.