# Architecting offline-first generative AI applications for edge deployments using AWS services

[AWS](https://yomu.fyi/company/aws) · Ahmed Ewis · Jul 22, 2026

**Type:** Problem & solution

## Summary

Industrial edge environments often face significant unplanned downtime yet lack reliable cloud connectivity to run large-scale artificial intelligence models. To address this limitation, an offline-first reference architecture shifts generative inference to edge devices while leveraging cloud services for preparation and lifecycle management. The workflow uses Amazon Bedrock with Amazon Nova Pro to create labeled dataset pairs, Amazon SageMaker AI Pipelines to fine-tune a compact model, and AWS IoT Greengrass to orchestrate edge deployments. Local execution relies on Ollama, Strands Agents, and a CPU-based ChromaDB retrieval pipeline that consumes zero GPU VRAM. In evaluations across three foundation model judges, the fine-tuned gpt-oss-20b model combined with retrieval-augmented generation consistently outperformed the base model configuration across accuracy, completeness, and relevance metrics.

## Context

Industrial facilities, offshore platforms, and remote agricultural sites suffer costly downtime but lack the reliable cloud connectivity needed to run standard cloud-based generative AI workloads.

## Approach / What changed

Deploy an offline-first hybrid architecture combining fine-tuning and retrieval-augmented generation. The system generates labeled training pairs with Amazon Nova Pro on Bedrock, fine-tunes gpt-oss-20b using SageMaker AI Pipelines, deploys artifacts via AWS IoT Greengrass, and executes inference locally using Ollama, Strands Agents, and a CPU-hosted ChromaDB vector store.

## Takeaways

- Running ChromaDB and a 384-dimension sentence-transformer on CPU and SSD keeps retrieval latency under 50 ms for datasets up to 5 GB without using GPU VRAM.
- Evaluating fine-tuned gpt-oss-20b with RAG against the base model showed average score increases across Claude 4.5 Haiku (85% vs 68.3%), Claude 4.5 Sonnet (76.7% vs 61.7%), and Nova Pro (82.5% vs 72.5%).
- AWS IoT Greengrass manages model packaging, versioning, and deployment without requiring persistent network connectivity to edge devices.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Deployment](https://yomu.fyi/topic/deployment), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://aws.amazon.com/blogs/architecture/architecting-offline-first-generative-ai-applications-for-edge-deployments-using-aws-services)