Loading…
Topic
Terraform
21 posts about Terraform. Every summary links to the original.
- 21 posts
- 12 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Updates on our pledge to make Cloudflare features accessible to everyone
Cloudflare CTO Dane Knecht pledged a year ago to make every Cloudflare feature available to all customers, not just Enterprise. The company has since expanded Logpush and Logpush Transformers, which use SQL to filter, redact, and reformat logs, to Free, Pro, and Business plans. Custom Dashboards for personalized metrics are now available to all customers. Role-Based Access Control (RBAC) has been extended to nearly all products, including resource-level roles for Workers, and a New Account button lets users create additional accounts directly. Organizations, which group accounts for unified analytics and shared configurations, are in beta for Enterprise and will reach free accounts in early 2027, and platform limits have also been raised, including Workers subrequests from 50 to 100 and Durable Objects idle time from 70–140 seconds to 15 minutes with active outbound connections.
How CSIRO built scalable, cost-optimized genomic variant querying on AWS
CSIRO built Serverless Beacon (sBeacon), a production-ready implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon protocol for secure genomic variant querying. Its serverless architecture combines Amazon S3, AWS Lambda, Amazon DynamoDB, and Amazon Athena: genomic data remains in its original S3 location, while metadata is indexed in ORC tables and ontology indexes for query processing. Standard VCF files can be queried through API Gateway, with modular Lambda functions resolving ontology terms, querying Athena, and, when needed, issuing HTTP byte-range requests to source buckets so raw VCF bytes do not leave their owning organization. The post reports support for hundreds of millions of individuals and billions of genomic locations, approximately USD 0.40 monthly operation for a 1000 Genomes-scale dataset, roughly five-second real-world queries, and about 18-second generation-to-use onboarding, while noting that whole-genome range queries can time out.
Protect Performance and Reduce Surprise Costs with Default Warehouse
Databricks SQL’s generally available Default Warehouse lets administrators choose the SQL warehouse automatically used for ad hoc experiences, addressing manual selection and fallback behavior such as “Last Selected” or alphabetical ordering. A workspace-level setting applies across SQL Editor, Catalog Explorer, AI/BI Dashboards, Alerts, and Genie Spaces, while users can override it or retain “Last Selected”; admin APIs and a Terraform resource support per-user assignment, auditing, and control. The feature supports serverless, pro, and classic SQL warehouses and respects existing governance and access controls, with the setting opt-in and new assets automatically using it. In customer results, Catalog Explorer queries sent to smaller warehouses rose from 77% to 96%, and users running ad hoc SQL across multiple warehouses fell by 15%, while future coverage is planned for MLflow, Lakeflow CLI, and DBSQL MCP.
MySQL CDC connector for ClickPipes is now Generally Available
MySQL CDC connector for ClickPipes is now Generally Available, enabling replication from MySQL and MariaDB into ClickHouse Cloud for analytical workloads. The release adds automatic parallel snapshotting: ClickPipes selects a logical partition key from a table’s primary key, including string and UUID types, and distributed processing reduced a benchmarked 1TB snapshot from over a day on one thread to about an hour across 32 workers. Production safeguards now require GTID-based replication and at least 72 hours of binlog retention, while new CPU, memory, network, and source, destination, and end-to-end lag metrics improve troubleshooting. MySQL CDC ClickPipes also support OpenAPI and Terraform 3.14.0+, are available across all ClickHouse Cloud service tiers, and began billing on September 1, 2026.
How the FDA is building a secure, AI-ready data foundation on Databricks for Government
The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.
MongoDB Atlas Joins Google Cloud Foundation Fabric (FAST): Secure Private Data Infrastructure in Minutes
Many organizations lack cloud fundamentals, making foundational setup complex and time-consuming; Google Cloud Foundation Fabric (FAST) is a GitOps-driven landing zone toolkit that accelerates establishing a production-ready Google Cloud organization. A new MongoDB Atlas module within FAST enables provisioning MongoDB Atlas clusters with Private Service Connect (PSC) for secure, private connectivity. PSC avoids VPC peering issues like CIDR conflicts and transitive routing, using a unidirectional consumer-producer model that respects project boundaries and routes traffic over Google’s private fiber network. The module automates cluster creation, PSC endpoint registration with Service Directory, and internal DNS resolution, integrating with Google Cloud KMS for customer-managed encryption keys. This integration reduces foundational setup from weeks to minutes, providing a secure data foundation that connects to BigQuery for real-time analytics and Vertex AI for GenAI workloads.
Provisioning for the Agentic Era: How Databricks Built a Self-Serve Infrastructure Vending Machine
Databricks built the Field Engineering Vending Machine (FEVM) to replace shared environments with isolated, centrally governed infrastructure for its growing field organization. Users or agents describe a use case and configuration details, receiving an environment assembled through a React frontend, Python backend, Databricks Apps, Terraform, Git Runner, and Lakebase across AWS, Azure, and GCP. FEVM supports templates and add-ons such as Lakebase, notebooks, and pre-packaged assets from UC Volumes, applies configurable time-to-live policies, and sends Slack notifications for provisioning, expiration, and deletion. It also manages shared resources independently, including catalogs that persist across workspace lifecycles, and provides administrative controls for limits and audits. At the time described, it managed more than 2,600 active deployments across three clouds and over 5,000 active users, after handling nearly 1,200 requests during one internal event.
Securing our codebase with autonomous agents
After a fivefold increase in pull-request velocity over nine months, the team found static analysis and rigid code ownership insufficient for security at scale. It built a fleet of Cursor Automations powered by cloud agents, webhook and GitHub integrations, and a security MCP deployed as a just-in-time serverless Lambda function. The MCP provides persistent data, semantic deduplication using Gemini Flash 2.5, and consistently formatted reporting and follow-up actions, while Terraform standardizes review and deployment. Four templates cover PR review, segmented vulnerability hunting, dependency patching with reachability analysis and tests, and daily invariant-drift monitoring. Agentic Security Review has run on thousands of PRs in two months and prevented hundreds of issues from reaching production; the team says more automations are planned.
Find the intersection of security, AI, IAM, and fun at Oktane
Oktane 2025 will focus on security, AI, and identity, with programming for protecting software systems that use both human and non-human identities. The event is scheduled for September 24–26, 2025, at Caesars Forum in Las Vegas, and will include the Oktane Developer Lounge, Expo Hall activities, lightning talks, documentation feedback games, and Developer Support consultations. Hands-on labs cover securing enterprise AI with the OAuth Cross App Access (XAA) extension and Model Context Protocol (MCP), automating Okta with Terraform, replacing standing administrative access with time-bound requests through Okta Identity Governance and Workflows, and using OAuth 2.0 Demonstrating Proof of Possession (DPoP) tokens in a single-page application. An Okta Workflows community meetup will add networking, drinks, and appetizers, while the broader event offers learning, expert conversations, and application-focused discussions.
Changes Are Coming to the Okta Developer Edition Organizations
Okta is replacing Developer Edition organizations with the Integrator Free Plan as the default developer organization type on May 22, 2025, as part of architectural changes tied to its Secure Identity Commitment. The new organization is intended for iterative integration development, identity-flow validation, and OIN preparation, not production use, and provides access to expanded testing offerings and support resources. Existing Developer Edition organizations begin deactivation on July 18, 2025, in batches, after which users lose access to their organizations and resources. Migration choices include recreating resources with automation, generating Terraform configurations with Terraformer, or using EnvSync to back up supported objects and recreate many of them. Published OIN integrations remain available, but linking them to the new account requires contacting Okta; draft or in-review submissions from Developer Edition must be approved by June 9 or will be automatically declined and require resubmission.
Supercharging LLM application development with LLM-Kit
Grab developed LLM-Kit, a comprehensive framework for setting up production-ready Generative AI applications while addressing security, scalability, integration, monitoring, and standardisation concerns. Its pre-configured structure includes an API server, configuration management, a sample LLM Agent, tests, and an integrated stack spanning Poetry, Gunicorn, FastAPI, LangChain, LangSmith, HashiCorp Vault, Amazon EKS, and GitLab CI. Engineers submit application details through a form, which triggers GitLab project and code-scaffold generation, CI/CD setup, Terraform-based staging and production infrastructure provisioning, and secure GPT-token storage in Vault. The scaffold also includes authentication, API documentation, observability, Redis and vector-database integrations, deployment pipelines, and a cookbook with organisational examples; the post reports that it has onboarded hundreds of GenAI applications and saves an estimated 1.5 weeks before teams begin their first feature.
How to Create a Secure CI/CD Pipeline Using Okta Terraform
The guide presents a secure CI/CD pipeline for managing Okta resources with Terraform, AWS, and GitHub Actions, addressing local setup, collaboration, state storage, secrets, and deployment automation. Its architecture uses GitHub as source control and workflow trigger, GitHub OIDC to authenticate to AWS IAM STS and assume roles, S3 for Terraform state, DynamoDB for state locking, and AWS Secrets Manager for Okta OAuth 2.0 client credentials private keys. Developers push changes to develop for an automatic development build, open a pull request to main for a production plan, and merge approved changes to trigger the production build. The workflows run Terraform init, plan, and apply as appropriate, while the guide also covers repository setup, branch organization, and troubleshooting for missed configuration in GitHub, Okta, or AWS.
Finding the right balance of speed and security through just-in-time access to cloud resources
Ramp’s rapid growth exposed problems in its AWS access model: three broad roles created persistent production access for some senior engineers while junior engineers depended on seniors for queries. The team first reorganized permissions into more than 20 team-specific AWS Identity Center groups, each backed by permission sets and a Terraform module that presents IAM configuration through human-readable resource lists. It then introduced just-in-time access through ConductorOne by pairing each non-production role with an elevated production role, removing standing production access and provisioning entitlements through AWS SSO group membership. Team-owned requests were automatically approved, while cross-team requests went to the relevant tech lead, with temporary access and backup approval handling described in the design. The resulting model reduced exposure, preserved fast access to needed resources, and recorded production activity for accountability and audit purposes.
Enterprise Maturity Workshop: Terraform
This workshop teaches how to use Okta’s Terraform provider to manage users and groups in an Okta Organization while practicing beginner and intermediate Terraform skills. It explains desired-state configuration, where terraform plan predicts changes and terraform apply makes them, and describes how providers, resources, variables, data sources, and the terraform.tfstate file fit into a project. The exercises cover provider setup with a client ID and private key, secret handling, creating and assigning users and groups, importing resources, refactoring files, formatting code, and controlling API usage. It also uses errors and configuration drift to show how Terraform and Okta can disagree, then compares importing manual changes with applying Terraform’s declared state. The workshop concludes with cleanup through terraform destroy and points to further provider and Terraform documentation.
Supporting Devs Through Advocacy
Semona describes her transition from Okta Developer Support Engineer to Developer Advocate after three years supporting admins, developers, Independent Software Vendors, and partners integrating with Okta workforce products. Her earlier work included OpenID Connect, OAuth 2.0, external and social identity providers, SCIM, OIN App Submission, Okta SDKs, the Okta Sign-in Widget, Token Inline Hooks, and Terraform, while cross-functional alliances connected her with Customer Success, Product, Engineering, Solutions Engineering, and Architects. She saw support cases as a way to understand developer problems and collect feedback that could inform improvements. In advocacy, she plans to use that experience to help developers avoid common problems, build community, learn with developers, and advocate for Okta developer community needs, with Python as her favorite programming language.
Scale-to-Zero Minecraft server with Terraform and Fly Machines
This tutorial presents a scale-to-zero Minecraft Java server built with Terraform and Fly Machines. It addresses the cost and resource burden of keeping a multiplayer server running when nobody is playing, while allowing it to restart when a player connects. The configuration provisions a Fly.io app, a Firecracker-based machine using Geoff Bourne’s itzg/minecraft-server Docker image, a 15GB persistent Fly Volume, an IPv4 address, and exposed TCP port 25565. Environment variables enable Autostop with estimated and initial timeouts of 120 seconds, while the machine receives four shared vCPUs and 8GB of RAM and mounts world data at /data. After terraform apply, the machine shuts down after the timeout and automatically starts again when Minecraft reconnects, though any TCP traffic can wake it and incur costs.
Logbook - 2022-06-01
Fly.io’s June 1, 2022 logbook records a week of platform, tooling, documentation, and dashboard changes, with forum community feedback driving several logged updates. The release adds rolling behavior to fly restart, extends fly status --all’s failed-VM visibility from two days to seven, supplies Laravel and Node.js examples, and introduces the first official Terraform provider with Machines API parity. Release commands receive 512MB of swap without increasing normal app RAM, while preview GraphQL endpoints fetch volumes, IPs, and certificates directly by ID to support Terraform imports and stateful tooling. Dashboard work adds Livebook upgrades, clearer billing and typography, individual Machine event tables, and fixes for IPv6 copying, organization-plan changes, cross-tab logout, billing URLs, misleading volume usage, and two-factor input limits.
Using DNS Traffic Management to Add Resiliency to Shopify’s Services
Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.
Building and Scaling a Global Chatbot using Heroku + Terraform
Live Agent, Salesforce’s embeddable chatbot, must handle customer interactions and requests at global scale while remaining highly available across regions. Its eight-person team chose Heroku to avoid managing bare metal, cloud virtual machines, databases, and other third-party services, while relying on managed Java support and regional infrastructure. The platform runs ten apps with Heroku Postgres, Redis, and Apache Kafka dependencies, and deploys them inside Heroku Private Spaces to isolate network boundaries and data. Terraform defines and replicates apps, add-ons, custom domains, logging, and profiling configuration across regions, allowing changes to propagate through a pull request. The result described is a simpler operational model in which Heroku handles maintenance and the team can scale the same architecture into new geographic locations.
Six Strategies for Deploying to Heroku
Heroku offers six deployment strategies: Git push, GitHub integration, Review Apps, Docker, Terraform, and the “Deploy to Heroku” button. Each has different requirements and trade-offs, and strategies can be combined in a workflow. Git push is simple but requires a developer with access to both repository and app, while GitHub integration can automate deployments and block them until CI tests pass. Review Apps provide disposable instances for pull requests; Docker supplies self-contained environments but requires maintaining a stack and does not support pipeline promotions. Terraform uses declarative HCL to manage Heroku and external infrastructure despite lacking official Heroku support, and buttons offer one-click deployments from GitHub repositories but do not auto-update or connect deployed apps back to their repositories.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.