Success Story
AWS data lake architecture for Kilimo: Climatic pipeline, S3/Athena partitioning and decoupling the ET0 calculation
Kilimo is an Argentine climatech and sustainability company that processes large volumes of climate data (multi-source scrapers, evapotranspiration calculation) on an AWS stack with EKS, RDS, S3, Glue, Athena and Airflow. Craftech has supported it since March 2026 with an AWS-funded data-engineering project.
Challenge
Kilimo needed to organize its data platform: move data from its Climatic system into a cost-efficient data lake, optimize queries, and decouple the evapotranspiration (ET0) calculation — a ~1,000-line script tied to the Django monolith — to run as an independent job. It also had friction with Airflow that warranted evaluating alternatives.
Solution
Craftech designed the data lake architecture: networking and IAM for Data Lake → RDS Climatic connectivity (EKS↔RDS, S3 VPC Endpoint), an S3 partitioning PoC (year/month/day/origin) with AWS Glue Catalog and Athena partition-pruning validation to reduce scanned data and cost, and bucket preparation to receive Climatic data. In parallel it audited the ET0 script to decouple it from the Django ORM and dockerize it as an Airflow job (KubernetesPodOperator), and evaluated orchestration alternatives.
Results
Kilimo has a validated data lake architecture (efficient S3 partitioning, Glue catalog and Athena queries optimized with partition pruning), resolved connectivity and permissions across Climatic, RDS and S3, and a clear plan to decouple the ET0 calculation from the monolith into an independent job. The work lays the foundation for a phase 2 (Sat and Hydromaps) and possible MSP continuity.
Stack & technologies
