Architecting offline-first generative AI applications for edge deployments using AWS services

Summary

Industrial edge environments often face significant unplanned downtime yet lack reliable cloud connectivity to run large-scale artificial intelligence models. To address this limitation, an offline-first reference architecture shifts generative inference to edge devices while leveraging cloud services for preparation and lifecycle management. The workflow uses Amazon Bedrock with Amazon Nova Pro to create labeled dataset pairs, Amazon SageMaker AI Pipelines to fine-tune a compact model, and AWS IoT Greengrass to orchestrate edge deployments. Local execution relies on Ollama, Strands Agents, and a CPU-based ChromaDB retrieval pipeline that consumes zero GPU VRAM. In evaluations across three foundation model judges, the fine-tuned gpt-oss-20b model combined with retrieval-augmented generation consistently outperformed the base model configuration across accuracy, completeness, and relevance metrics.

Context

Industrial facilities, offshore platforms, and remote agricultural sites suffer costly downtime but lack the reliable cloud connectivity needed to run standard cloud-based generative AI workloads.

Approach / What changed

Deploy an offline-first hybrid architecture combining fine-tuning and retrieval-augmented generation. The system generates labeled training pairs with Amazon Nova Pro on Bedrock, fine-tunes gpt-oss-20b using SageMaker AI Pipelines, deploys artifacts via AWS IoT Greengrass, and executes inference locally using Ollama, Strands Agents, and a CPU-hosted ChromaDB vector store.

Takeaways

• Running ChromaDB and a 384-dimension sentence-transformer on CPU and SSD keeps retrieval latency under 50 ms for datasets up to 5 GB without using GPU VRAM.
• Evaluating fine-tuned gpt-oss-20b with RAG against the base model showed average score increases across Claude 4.5 Haiku (85% vs 68.3%), Claude 4.5 Sonnet (76.7% vs 61.7%), and Nova Pro (82.5% vs 72.5%).
• AWS IoT Greengrass manages model packaging, versioning, and deployment without requiring persistent network connectivity to edge devices.

Related reading

• AWS · Aug 26, 2026

  Gallup scales real-time coaching for thousands with Amazon Bedrock

  Gallup needed to centralize decades of proprietary workplace performance research to provide proactive, real-time coaching for organizational leaders directly within their daily workflows. To solve this challenge, Gallup developed Gallup AI inside its Gallup Access platform using Amazon Bedrock and Anthropic Claude models. The multi-tenant serverless architecture leverages Amazon Bedrock Knowledge Bases and Amazon Kendra for retrieval-augmented generation, while AWS Lambda with FastAPI powers real-time streaming responses. Amazon ElastiCache Serverless caches recent conversation history for sub-millisecond session retrieval, Amazon RDS for MySQL provides durable transactional storage, and Amazon Bedrock Guardrails enforce content safety. Since its June 2024 launch, the system has achieved sub-second time-to-first-byte streaming latency while prompt volume grew approximately sevenfold across thousands of organizations.

  Tamil Sambasivam

• AWS · Aug 13, 2026