Loading post…
AWSProf. Denis BauerExplainer
How CSIRO built scalable, cost-optimized genomic variant querying on AWS
Summary
CSIRO built Serverless Beacon (sBeacon), a production-ready implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon protocol for secure genomic variant querying. Its serverless architecture combines Amazon S3, AWS Lambda, Amazon DynamoDB, and Amazon Athena: genomic data remains in its original S3 location, while metadata is indexed in ORC tables and ontology indexes for query processing. Standard VCF files can be queried through API Gateway, with modular Lambda functions resolving ontology terms, querying Athena, and, when needed, issuing HTTP byte-range requests to source buckets so raw VCF bytes do not leave their owning organization. The post reports support for hundreds of millions of individuals and billions of genomic locations, approximately USD 0.40 monthly operation for a 1000 Genomes-scale dataset, roughly five-second real-world queries, and about 18-second generation-to-use onboarding, while noting that whole-genome range queries can time out.
Context
sBeacon applies the GA4GH Beacon protocol to genomic and phenotypic data discovery. The implementation is intended for clinical and research applications, including federated networks where data custodians retain control of sensitive genomic data.
Approach / What changed
sBeacon uses Terraform for deployment and combines API Gateway, Lambda, S3, DynamoDB, Athena, and ontology indexing. Metadata is loaded and indexed in ORC format, while genomic data is referenced in place; query functions resolve ontology descendants, query metadata, and use htslib HTTP byte-range requests against source S3 buckets when variant details are needed.
Takeaways
Genomic data is not copied into sBeacon: queries use HTTP byte-range requests against the owning organization’s S3 bucket, and only an exists, count, or variant-record result is returned.
The reported operating cost is approximately USD 0.40 per month for a 1000 Genomes-scale dataset, with real-world query responses taking about five seconds.
Whole-genome range queries can time out at the API Gateway level, and fan-out queries require monitoring Lambda concurrency and throttle metrics to avoid capacity pressure or lost results.
FAQ
How does sBeacon keep raw genomic data under the source organization’s control?
The query path uses htslib to issue HTTP byte-range requests against the Amazon S3 REST API of the organization owning the bucket. Raw VCF bytes stay in that bucket; only the query result—exists, count, or variant record—is returned.
Who is allowed to onboard datasets into sBeacon?
Dataset onboarding requires authentication through the Amazon Cognito authorizer and membership in the sbeacon-admin-group. Users with only record-access or count-access are rejected, so onboarding is not self-service.
What operational issues can affect burst queries?
A fan-out query can consume substantial Lambda concurrency, and synchronous invocation does not automatically retry after throttling. Cold starts can also add latency after idle periods; provisioned concurrency reduces that latency but increases idle cost.
Source and freshness
- Source
- aws.amazon.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.