Posts tagged coiled

Coiled observability wins: Chunksize

Distributed computing is hard, distributed debugging is even harder. Dask tries to simplify this process as much as possible. Coiled adds additional observability features for your Dask clusters and processes them to help users understand their workflows better.


Read more ...

Parallel Serverless Functions at Scale

The cloud offers amazing scale, but it can be difficult for Python data developers to use. This post walks through how to use Coiled Functions to run your existing code in parallel on the cloud with minimal code changes.

Comparing code runtime between a laptop, single cloud VM, and multiple cloud VMs in parallel

Read more ...

Reduce training time for CPU intensive models with scikit-learn and Coiled Functions

You can use Coiled Run and Coiled Functions for easily running scripts and functions on a VM in the cloud.

Code snippet adding coiled.function decorator to scikit-learn model training.

Read more ...

Fine Performance Metrics and Spans

While it’s trivial to measure the end-to-end runtime of a Dask workload, the next logical step - breaking down this time to understand if it could be faster - has historically been a much more arduous task that required a lot of intuition and legwork, for novice and expert users alike. We wanted to change that.

Populated Fine Performance Metrics dashboard

Read more ...

Data-proximate Computing with Coiled Functions

Coiled Functions make it easy to improve performance and reduce costs by moving your computations next to your cloud data.


Read more ...

Process Hundreds of GB of Data with DuckDB in the Cloud

DuckDB is great tool for running efficient queries on large datasets. When you want cloud data proximity or need more RAM, Coiled makes it easy to run your Python function in the cloud. In this post we’ll use Coiled Functions to process the 150 GB Uber-Lyft dataset on a single machine with DuckDB.

Code snippet of using the coiled.function decorator to run a query with DuckDB on a large VM in the cloud.

Read more ...

How to Train a Neural Network on a GPU in the Cloud with coiled functions

We recently pushed out two new and experimental features coiled run and coiled functions which is a deviation of coiled run. We are excited about both of them because they:

Read more ...

Dask performance benchmarking put to the test: Fixing a pandas bottleneck

Getting notified of a significant performance regression the day before release sucks, but quickly identifying and resolving it feels great!

Read more ...

Coiled notebooks

We recently pushed out a new, experimental notebooks feature for easily launching Jupyter servers in the cloud from your local machine. We’re excited about Coiled notebooks because they:

Read more ...

Observability for Distributed Computing with Dask

Debugging is hard. Distributed debugging is hell.

When dealing with unexpected issues in a distributed system, you need to understand what and why it happened, how interactions between individual pieces contributed to the problems, and how to avoid them in the future. In other words, you need observability. This article explains what observability is, how Dask implements it, what pain points remain, and how Coiled helps you overcome these.

The Coiled metrics dashboard provides observability into a Dask cluster and its workloads.

Read more ...

Performance testing at Coiled

At Coiled we develop Dask and automatically deploy it to large clusters of cloud workers (sometimes 1000+ EC2 instances at once!). In order to avoid surprises when we publish a new release, Dask needs to be covered by a comprehensive battery of tests — both for functionality and performance.

Nightly tests report

Read more ...