preloader
  • Hello Cerebas

Recently, the two CS-3 systems by Cerebras have been made available to users at the Helmut Schmidt University. This tutorial aims to simplify getting started with using this specialized hardware for HPC applications.

blog-thumb
Image by "Markus Stroopwafels" via Wikimedia Commons

Introduction

The Dutch delicacy Stroopwafel is a pretty good visual analogy for the Wafer-Scale Engines that run inside the two Cerebras CS-3 systems that the Helmut Schmidt University received earlier this year:
Both are made using the largest possible form factor, both are divided into a regular grid, and both deliciously accelerate scientific computing output.

This gives the WSE hundreds of thousands of Processing Elements . In fact, every job on a single CS-3 allocates the full wafer, no matter how many PEs make up the actual program rectangle.

Let’s compare the CS-3 to other parallel systems:

SystemAvailable coresDetails
CS-3883350Usable program rectangle of 755$\times$1170 PEs (without halo)
HSUper40968569 nodes $\times$ 72 cores on the largest partition1$^,$2
GPUfew tens of thousands3CUDA cores are simpler than CPU cores and non-independent/coupled through SIMT .

However, leveraging this massive parallel processor for HPC is not trivial, since the program must be written in Python and the Cerebras Software Language 4. (Another main focus of the CS-3 is training ML models, but this topic is not covered in this article.) To expedite the productive use of this hardware, I have put together a simple boilerplate project as an addition to the official SDK examples on GitHub, which can also be used with the Simulator on HSUper. The example illustrates using the WSE for an embarrasingly parallel problem and using the standard library for collective communication across the PEs .

Prerequisites

Before we begin, let’s make sure that you can work with the Cerebras simulator or device.

To use the simulator
make an interactive HSUper allocation and run ml python cerebras module.
(For working on another system, such as your own laptop, install the Cerebras SDK.)

To use the device
connect to a Cerebras login node over SSH as described here. Then install uv for running Python scripts. (This ensures installation of the required Python packages, cerebras-appliance and cerebras-sdk version 2.10.0, is handled automatically.)

Download the example (Note: This can only be accessed through the HSU network or the VPN.):
git clone https://git.hpc.hsu-hh.de/rubenhorn/hello-cerebras

The Example

The code in the example repository performs are simple map-style operation where for each PE , we compute $y = i_{\text{PE}} + b$ with the global id of the PE $i_{\text{PE}} \in I$ where $I$ = {$y \cdot \text{width} + x$ | $(x,y) \in [0,\text{width}-1] \times [0,\text{height}-1]$}. For such embarrassingly parallel workloads, using the WSE is as straight forward as using a GPU .

The steps involved are illustrated in the diagram below:

5.Copy1.rePsr3ue.lptaUsrseefric4ono.mplulRdteuecnvwtiiiectvmehebathcrooorsmahtmsouiscnntoigdcleaytipoanratl6ol.e2blV.reokrCaeiodrfpcnyyae/slritenadpicuunrctpoeusftsrreoaPsmcEurslhotosssstwPitEtoshdtehveicheostcode
  1. Usually, one might need to start by loading the input data on the host side. For simplicity, the only input parameter is hardcoded as $b=1$.
  2. Next, this input is transferred to the PE of the device or simulator with $i_\text{PE}=0$.
  3. This value is then broadcast across all PEs along the x- and y-axis.
  4. The computation is performed in parallel across all 883350 PEs (16 on the simulator).
  5. Finally, resulting 883350 (or 16) values are copied back to the host
  6. and compared to a NumPy-implementation.

This trivial example is just meant as an easy to understand example of the components of a Cerebras HPC code. However, all the discussed components may be used to achieve some actually interesting computation, such as a gravitational $N$-body simulation5: Broadcasting the position of planets along the x- and y-axis of the program rectangle results in each PE computing a single pairwise force contribution. X-axis reduction by summation and integration can likewise be performed using high-level collective operations from the collectives_2d-library.

Compile and run the example using:

1
2
3
4
TARGET=local_sim # On HSUper
TARGET=app_dev # On Cerebras login
./compile.py -t $TARGET
./run.py -t $TARGET

Technical Details

With that, let’s look at the technical parts of the example.

Custom SDK-Wrapper

Copying memory to and from the device and calling device functions require the corresponding SDK object. To simplify these operations and to allow seamless use of the same scripts on both the simulator and the device, I wrote the simple_wse.py module containing wrapper code around the Cerebras SDK . This module is included with the example project and the script run.py uses it as described below. All subsequent information thus relates to this helper code and does not necessarily match the underlying usage in the Cerebras SDK one-to-one. The main class is the CerebrasWrapper which is used as a context manager to interact with the simulator or allocate the device in appliance mode:

1
2
3
4
5
with CerebrasWrapper(
    artifact_path, # Path to the compiled WSE program ("out"-folder or tar ball)
    target # "local_sim" or "app_dev"
) as cw:
  # memcpy_h2d, launch, memcpy_d2h
The lifetime of this context manager must match the lifetime of your code’s globally device memory. Also, acquiring the device in appliance mode takes several seconds, even when there are currently no other jobs running.

Data is transferred to the simulator/device in the form of three dimensional NumPy-arrays where the first two dimensions match the corresponding row and column of the program rectangle PE . As described above, a single scalar is copied to a single PE in the example:

1
2
3
4
b_pe = np.array([b], dtype=np.uint32) # Values per PE where b = 1 (scalar)
b_h2d = b_pe[np.newaxis, np.newaxis, :] # Reshape to (PE_x, PE_y, n)
# Copy data with shape (1,1,1) and no offset (px/py) => PE_(0,0)
cw.memcpy_h2d(name="b", values=b_h2d)
The same holds for transferring data from the simulator/device:
1
2
3
4
5
6
7
Y = cw.memcpy_d2h(
    name="y",
    width=width,
    height=height,
    n=1, # One scalar result per PE
    dtype=np.uint32,
).squeeze(-1)  # Delete last dimension, since it's a scalar
The actual computation on the device/simulator is executing using:
1
cw.launch("main") # Blocks until completion

Compiler-Helper for Any Program

The compilation script compile.py likely does not need to be changed for compiling other codes. Parameterization of the programming rectangle and the single PE kernel code are done using get_PE_rect(use_simulator) and get_params(use_simulator) from params.py, respectively.

For example, to add a compile-time scaling factor alpha, add the following lines:

FileContents
params.py
1
2
# in def get_params(...)
params["alpha"] = 10 # or some other value
layout.csl
1
2
3
4
param alpha: u32;
// ...
// just below @set_tile_code(...
.alpha = alpha,
pe_program.csl
1
2
3
4
param alpha: u32;
// ...
// Replace line in task compute() void
y = alpha * pe_id + b;

The role of the other two files is explained in the following section.

Anatomy of the Device Code

Cerebras device code is divided into two parts: layout and PE code.

FileContentsRole
layout.cslLayout codeMap PE code to the program rectangle
pe_program.cslPE codePer- PE kernel code

The layout is used to define the programming rectangle, perform global compile time calculations/checks, export variable/function names to the host, and map kernel code and parameters to the individual PEs :

1
2
3
4
5
6
7
8
@export_name("y", [*]u32, false);
@export_name("main", fn()void);
// ...
@set_rectangle(width, height);
// ...
@set_tile_code(x, y, "pe_program.csl", .{
  // Kernel parameters
});
Note that excessive variation of the PE source code or parameter values results in longer compilation time and larger artifacts. For example, you should NOT pass x and y as parameters into the kernel but look them up in pe_program.csl using the <layout>-library instead.

In the PE code, the variables and functions are exported to match the names exported in the layout code. Note that pointers to the arrays holding the actual data are exported:

1
2
3
4
5
6
7
var y_arr: [1]u32; // Only array-pointers can be used with memcpy
const y_ptr: [*]u32 = &y_arr;
// ...
comptime {
  @export_symbol(y_ptr, "y");
  @export_symbol(main);
}

In addition to conventional, synchronous functions, tasks are used to perform computation as a response to communication across the fabric, asynchronously. The tasks comprising the example are shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
fn main() void {
  // ...
  @activate(bcast_x_task_id);
}
//      |
//      v
task bcast_x() void {
  mpi_x.init();
  // Parameters: root x, send/recv buffer, num elements, callback
  mpi_x.broadcast(0, b_ptr, b_size, bcast_y_task_id);
}
//      |
//      v
task bcast_y() void {
  mpi_y.init();
  mpi_y.broadcast(0, b_ptr, b_size, compute_task_id);
}
//      |
//      v
task compute() void {
  const b = b_arr[0];
  y = pe_id + b;
  @activate(exit_task_id);
}
//      |
//      v
task exit() void {
  // Make results visible to memcpy via array
  // (If necessary, make sure to (bit)cast non-u32 values)
  y_arr[0] = y;
  memcpy.unblock_cmd_stream();
}
The exit-task is essential to guarantee that the kernel completes cleanly.

The task code is linked to one of 23 task-ids (8-30) as so:

1
2
3
4
const compute_task_id: local_task_id = @get_local_task_id(17);
comptime {
  @bind_local_task(compute, compute_task_id);
}

Collective Communication Wiring

The 2d-collective communication uses Cerebras concepts such as colors and Data Structure Descriptors 6, to communicate data across the PEs , the details of which are outside of the scope of this article.

Broadcasting values along one axis requires two of the 24 routable colors and its two local task-ids must not collide with those used for the other tasks. The corresponding parameters are created in the layout code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
const c2d = @import_module("<collectives_2d/params>");

const c2d_x_color_0: color = @get_color(0);
const c2d_x_color_1: color = @get_color(1);

const c2d_x_entrypt_0: local_task_id = @get_local_task_id(10);
const c2d_x_entrypt_1: local_task_id = @get_local_task_id(11);
const c2d_params = c2d.get_params(x, y, .{
  .x_colors      = .{ c2d_x_color_0,   c2d_x_color_1 },
  .x_entrypoints = .{ c2d_x_entrypt_0, c2d_x_entrypt_1 },
});

These parameters are used for importing the module in the PE code, which allocates two out of the eight queues:

1
2
3
4
5
6
7
8
param c2d_params;
const mpi_x = @import_module("<collectives_2d/pe>", .{
  .dim_params = c2d_params.x, // From layout
  .queues = [2]u16{2,4},      // Queue ids
  .dest_dsr_ids = [1]u16{1},  // Data structure register id
  .src0_dsr_ids = [1]u16{1},  // Data structure register id
  .src1_dsr_ids = [1]u16{1},  // Data structure register id
});

Conclusion

Beyond efficient training of ML models, the CS-3 systems provide extremely parallel resources for general purpose computing. For map-and-reduce-like workloads, the provided example can be applied to speed up data processing tasks, where the programming model mirrors that of a GPU . Asynchronous, color-based communication between neighboring PEs and DSDs unlock the potential for more advanced computations, but also increases the programming complexity.


  1. HSUper Documentation – Technical Specifications ↩︎

  2. HSUper Documentation – Partitions ↩︎

  3. www.techpowerup.com, “GPU Database – NVIDIA L40S”, https://www.techpowerup.com/gpu-specs/l40s.c4173, n.d., (accessed Nov. 06, 2026). ↩︎

  4. Cerebras, “Cerebras SDK – Core Language”, https://sdk.cerebras.ai/csl/language, n.d., (accessed Nov. 06, 2026). ↩︎

  5. In the interest of brevity, this is left as an excercise for the reader. ↩︎

  6. Cerebras, “Cerebras SDK – Data & Execution”, https://sdk.cerebras.ai/csl/language/dsds, n.d., (accessed Nov. 06, 2026). ↩︎