Liqid is proud to have had the privilege to attend and present at Bio IT World 2021 in Boston, Massachusetts from September 20-22. Missed the amazing presentation by Liqid’s Chief AI Architect, Josiah Clark? Fear not, the whole talk is online, but we’ll do the honors of summing it up here on the Liqid blog as well.
Our awesome team in person at the BioIT conference discussed Liqid’s Composable Disaggregated Infrastructure (CDI) architecture and how it can be used for data processing in HPC and the computational science space. Need a refresher on CDI? Let’s start with the shortcomings of various IT infrastructure solutions.
Virtualization is one option which allows us to easily carve up resources. In a similar vein, containerization allows us to carve up large workloads to run different libraries on the same machine without interfering with another application’s configuration. Finally, On-Prem Infrastructure-as-a-Service (IaaS) allows the flexibility of cloud-like orchestration, but in person.
However, all these methods are limited to the physical infrastructure of a server. Put simply, CDI takes resources from different hosts and pulls them out of the servers physically and inserts them into separate pools for resources such as GPUs, network interconnects, and high-performance storage. When applied to the PCIe bus, this configuration allows us to compose on-demand servers using Liqid Matrix based on the job needed in the moment.
Liqid handily bypasses data bottlenecks using PCIe bus, leading to huge performance benefits which improve throughput and minimize latency. Let’s now look at how traditional architectures in computational biology and see if CDI can help.
Shared Storage
- Issues: fixed bandwidth and node, and generally monolithic upgrades for next-gen networks.
- CDI applied to shared storage limitations allows job-based HCA allocation as well as the ability to disruptively add next-gen HCAs.
Scratch Space
- Issues: jobs requiring the same datasets require costly and time-intensive job reads, as well as the inability to expand scratch size.
- CDI applied to scratch space limitations prevents an unnecessary secondary read-write operation for same dataset which saves time and resources. One can also dynamically expand scratch space based on certain job needs.
Data Sharding
- Issues: one isn’t able to move shards between node types, as well as the inability to increase shard capacity dynamically.
- CDI applied to data sharding remaps sharded datasets based on the compute node required for algorithms, additionally non-disruptively increasing local storage space based on growth needs.
Liqid is the better way to enable the biotech community to better solve computationally intensive problems. Watch the full talk by Josiah Clark, Chief AI Architect, here. Want to learn more about Liqid’s CDI solutions? Download our Composable White Paper here.
