7. Operations phase: Monitor / maintain
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed do eiusmod tempor incididunt ut labore et dolore magna.
Why should I do this?
Continuously monitor performance, safety, and real-world impacts, and maintain the system. Use feedback and new data to identify issues, make improvements, and ensure the system remains effective, safe and compliant over time.
- Workflow activities Miro Board
Target users
- CTOs and technical leaders
- Data scientists and ML engineers
- Product managers
- Research organizations, think tanks
Checklists in play
- Checklist 1: Risk Identification & Mitigation (Q18, Q19, Q20, Q21)
- Checklist 2: Participatory Co-Creation, Feedback, and Model Refinement (Post-deployment Q3, Post-deployment Q4, Post-deployment Q5, Post-deployment Q6, Post-deployment Q7)
- Checklist 3: Legal & Regulatory Compliance (Q15)
Questions to consider
Checklist 1 · Q18
How are performance metrics and model drift tracked continuously and responded to?
Checklist 1 · Q19
What patch management process for updates and security monitoring have been put in place?
Checklist 1 · Q20
What/who is responsible for ongoing, post-deployment accountability for the performance of the system?
Checklist 1 · Q21
How are data flows reviewed periodically for continued compliance?
Checklist 2 · Post-deployment Q3
Which training approach would be used to iteratively refine the model over time?
Checklist 2 · Post-deployment Q4
How will you assess that feedback from users has influenced the AI system's design, functionality, or intended use?
Checklist 2 · Post-deployment Q5
What triggers mandatory vs discretionary model refinement, version control requirements, and rollback procedures?
Checklist 2 · Post-deployment Q6
What safeguards are in place for post-hoc refinements to improve the model, enhance accuracy, and reduce errors or hallucinations?
Checklist 2 · Post-deployment Q7
What mechanism is available for documenting model iterations?
Checklist 3 · Q15
Is compliance periodically reassessed (including an assessment of emerging legislation, regulations, guidelines, advisories, or industry manuals) to account for regulatory or system changes?
Investment types
Every investment project is unique
The application of the six steps will vary accordingly. To provide examples that align with your project, common characteristics of AgDev investments were researched and three ‘investment types’ were developed.
AgriConnect: a digital solutions investment
User feedback, incident reports and updated agricultural guidance were continuously incorporated into future model refinements and dataset updates.
Participatory co-creation, feedback, and model refinement
- Are affected communities informed of major changes?
- Are lessons learned integrated into future refinements?
- Are feedback mechanisms still inclusive and active?
Lessons learned and challenges that arose
The team recognized that many risks only emerged gradually over time as agricultural conditions, user behavior and regulations evolved. Some recommendations became inappropriate following changes to local agricultural guidance.
Decisions taken and documentation maintained
Continuous monitoring processes were formalized through recurring governance reviews, stakeholder advisory meetings and periodic compliance reassessments. Retraining decisions, dataset updates and incident responses were documented in governance records and audit logs to maintain accountability and traceability over time.
AgroThrive: a policy and advocacy investment
Feedback, policy changes, new research and evolving agricultural priorities were incorporated into ongoing refinements.
Participatory co-creation, feedback, and model refinement
- Are affected stakeholder groups still represented in review processes?
- Are communities able to challenge harmful or misleading synthesis?
Lessons learned and challenges that arose
Over time, the team observed that repeated user interactions reinforced dominant framing patterns. Frequently requested narratives became increasingly prioritized by the system, creating a subtle feedback loop that narrowed diversity of policy perspectives.
Decisions taken and documentation maintained
The team introduced periodic audits reviewing whether outputs continued to represent a broad range of evidence and viewpoints. Audit findings, corrective actions and governance responses were documented in periodic monitoring reports and fairness review registers.
NGBT: a field research investment
User feedback, adoption outcomes and changing environmental conditions were incorporated into future refinements.
Participatory co-creation, feedback, and model refinement
- Are communities still represented in refinement discussions?
- Are communities who did not opt to use the new crop able to give feedback?
- Are negative experiences and unintended outcomes actively collected?
Lessons learned and challenges that arose
The team found that repeated optimization toward successful adoption metrics gradually reduced visibility of alternative farming systems and traditional resilience practices within generated responses.
Decisions taken and documentation maintained
The performance metrics were documented and to improve this metric, the team increased human oversight to actively monitor and flag overconfident responses.