<span class="translation_missing" title="translation missing: en.options.talk">talk</span>: How to build a federated open source serverless tensor data lakehouse for petabyte scale foundation model training

At IBM Research we are dealing with triple-digit petabyte scale earth observation and atmospheric physics data spread among different cloud and HPC data centers around the globe.

Based on open source and open standards we’ve build a tensor data lakehouse capable of streaming random tensors directly from large virtual super-cubes from disk to GPU memory on GPU clusters using state-of-the-art hardware.

We’ve used this system for training our latest generation of geospatial-temporal foundation models. We also explore use cases in neuroscience and genomics.

Info

Day: 2023-10-21
Start time: 15:40
Duration: 00:40
Room: HG D 1.1

Links:

Feedback

Click here to let us know how you liked this event.

Concurrent Events