MicroGenAI: Generative Language Models for Unearthing Resilient Microbes in Regenerative Agriculture

This repository contains the official implementation and experimental framework for my Master's Thesis in Data Science at FCUL - Universidade de Lisboa.

Academic Credits

• Student: Filipe Marques
• Supervisor: Francisco Couto
• Co-supervisor: Ana Margarida Fortes

Summary

The MicroGenAI framework addresses a critical bottleneck in agricultural data science: the manual and time-consuming process of curating scientific knowledge on plant-microbe interactions. By leveraging open-source Large Language Models (LLMs), this project implements an automated, scalable pipeline capable of extracting nuanced biological relationships from peer-reviewed literature.

The project specifically evaluates the capabilities of LLMs in extracting complex biological relationships involving plants, microorganisms, and environmental factors.

Relations

• Project: Microdrygrape Project @ FCUL

---

Features

• Multi-GPU Orchestration: Automated workload distribution across multiple GPUs using Hugging Face's device_map.
• Fault-Tolerant Pipeline: Equipped with automated exception handling and real-time checkpointing (STRICT vs VERBOSE logging) to safeguard long-running evaluation sessions.
• Automated Statistical Evaluation: Built-in module computes $Accuracy$, $Precision$, $Recall$, and $F_1\text{-Score}$.

---

How To Run

1. Environment Setup

We recommend using a Conda environment to manage dependencies:

conda create -n Name python=3.11
conda activate Name
pip install -r requirements.txt

2. Configuration

• Copy .env.example to a new file named .env.
• Add your Hugging Face access token to the .env file.

3. Pipeline Execution

nohup python -u main.py >> runs/terminal_log.txt 2>&1 &

4. Monitoring

Monitor the real-time inference progress via the log file:

tail -f runs/terminal_log.txt

---

Repository Structure

├── data/
│   └── MicroGenAI/
│       ├── Checked_DS            # Corpus Missing 2025/2026 inputs
│       ├── article_list          # Full dataset with all the articles
│       └── GTD.txt               # Ground Truth Dataset
│
├── src/
│   ├── __init__.py               # Package initializer
│   ├── config.py                 # Global constants, hyperparams & prompt template
│   ├── utils.py                  # Text normalization & statistical metrics
│   ├── model_loader.py           # Environment auditing & GPU weight allocation
│   └── pipeline.py               # Core inference loop and checkpoint manager
│
├── main.py                       # Central orchestrator entry point
├── metrics.py                    # Statistical evaluation engine
├── llm_review_pipeline.py        # Human-in-the-loop annotation and verification
├── .env.example                  # Template for secure environment variables
├── .gitignore                    # Specifies intentionally untracked files
└── requirements.txt              # Pinpointed library dependencies (CUDA 11.8 compatible)

---

To-Do List

• Add articles from 2025/2026 to the corpus.
• Incorporate additional validation datasets.

Funding

This work is funded by national funds through FCT – Fundação para a Ciência e a Tecnologia, I.P. under the LASIGE Research Unit, ref. UID/00408/2025, DOI https://doi.org/10.54499/UID/00408/2025, and LASIGE Seed project LASIGE-SP-2025.08