Here’s a small overview of what I learned.

Modeling Curves and Surfaces

In Assignment 1, I implemented cubic Bézier and B-Spline curves, conversion between their control-point representations, and bicubic tensor product patches. This gave me a better understanding of non-polygonal representations for surfaces. Since the patches are continuous and differentiable, we can easily get the normal for lighting as well.

Attaching a Mesh to a Skeleton

Assignment 2 covered hierarchical transforms and skeletal subspace deformation, also called linear blend skinning. I built a node hierarchy and attached a mesh using per-vertex joint weights. Rotating a parent joint carries its descendants with it, while the weights let a vertex respond to several joints at once.

The approach is to first move a vertex from the character’s bind pose into a joint’s bind-pose coordinate frame, then move it back using that joint’s current transform. I also passed accumulated transforms down the scene tree when collecting objects for rendering, avoiding repeated walks up the hierarchy. I believe this is very similar (if not identical) to how videogames handle object rendering trees.

glm::mat4x4 &bind_pose_joint_to_character_inverse_ = Bs_[j];
glm::mat4x4 &joint_to_character = joint_to_character_matrices[j];
glm::vec4 position_contribution = (attachment_weights_[i][j] *
joint_to_character * bind_pose_joint_to_character_inverse_ * bind_pose_position);

new_positions->at(i) += glm::vec3(position_contribution);

Simulating Motion with Particles and Springs

For Assignment 3, I implemented forward Euler, explicit trapezoidal, and fourth-order Runge–Kutta integration, then used them for circular motion, a spring-connected pendulum, and wireframe cloth. The particle system combines gravity, drag, and spring forces. The integrator then advances its positions and velocities over time.

The cloth is a grid with structural springs connecting neighbors, shear springs connecting diagonals, and flex springs spanning two grid edges. These resist stretching, diagonal collapse, and bending. This was more of a programming exercise, but it was still cool to see how to make a cloth simulation.

Rendering with Rays

We were tasked with building a CPU ray tracer. I implemented sphere, plane, and triangle intersections, Phong lighting, shadow rays, and recursive reflections. Rays are transformed into object space for intersection, while normals return to world space using the inverse transpose of the object’s transform.

if(bounces > 0) {
  // ray bounces
  const glm::vec3 ray_bounce_origin    = hit_position + EPSILON * eye_reflection;
  const glm::vec3 ray_bounce_direction = eye_reflection;
  const Ray       ray_bounce(ray_bounce_origin, ray_bounce_direction);
  HitRecord bounce_record = HitRecord();
  color += material_specular_color * TraceRay(ray_bounce, bounces - 1, bounce_record);
}

Casting Shadows in OpenGL

Assignment 5 involved texture mapping and shadows from a directional light. I rendered the scene from the light’s viewpoint into a depth texture, then sampled that texture while shading the camera’s view. If a fragment is farther from the light than the stored depth, another surface is blocking it.

This involved framebuffer objects, texture sampling, coordinate transforms, and GLSL. A small depth bias helps prevent a surface from incorrectly shadowing itself because of numerical error.

vec3 proj_coords = (world_to_light_ndc_matrix * vec4(world_position, 1.0)).xyz;
proj_coords = proj_coords * 0.5 + 0.5;
float this_depth     = proj_coords.z;
float occluder_depth = texture(shadow_texture, proj_coords.xy).r;

float bias = 0.005;
if(occluder_depth < this_depth - bias) {
    return vec3(0.0, 0.0, 0.0);
}

These assignments preceded my final project, emulating Ghost of Tsushima’s procedural grass in Godot.

Updating a singular boid’s position requires the normalization of several vectors, which presents the need to develop a resource-efficient and low-latency module for vector normalizations. To acquire the three-dimensional position of the target, we implemented stereo vision using a pair of FPGAs — each with their own camera — which communicate with each other through the SPI protocol. Using the target’s positional information, we computed the positions of the boids every frame in three dimensions and overlaid them on the camera footage via a double frame buffer.

The double frame buffer also presented a unique challenge since, due to overlaying the boids over the camera feed, there was a limited amount of additional pixels/DRAM requests that could be made every frame, amounting to roughly 25% of the screen’s area.

For this class project, we committed to having multiple boids follow the target, moving every frame and being rendered at their computed positions. To demonstrate proper depth calculation, we modified every boid’s size based on its depth. The implemented boid update module includes only the target-following force; alignment, cohesion, and separation remain future work.

Contributions

This was a final project for MIT’s 6.2050 Digital Systems Laboratory, built with Sreeja Thipireddy. I was responsible for the implementation of the normalizer, boid manager, and boid update modules. Sreeja was responsible for the implementation of the inter-FPGA communication protocol, depth calculation module, double frame buffer, and modifications to the traffic generator. The renderer module was implemented in collaboration.

Vector Normalization

To compute the necessary vectors in the boid computations, it is necessary to normalize vectors. To normalize a vector, it’s sufficient to multiply the vector’s components by the reciprocal of the vector’s magnitude.

For a vector $\mathbf{v} = (v_x, v_y, v_z)$, we first compute its squared magnitude:

This lets us compute one reciprocal square root and reuse it for all three components.

Initial Approximation

We can compute an initial approximation to the reciprocal of the square root via the position of the number’s most significant bit.

Using $x \approx 2^b$, we obtain the initial approximation

For a nonzero integer $x$, the position $b$ of its most significant bit is

I implemented a custom combinational module with a tree-based approach which computes the count of leading zeroes. The result for any n-bit number can be derived from the result of its left and right halves. If the upper half of the number contains a 1 bit, the result comes from that half. Otherwise, the result is the width of the upper half plus the leading-zero count of the lower half. This gives us a combinational tree for finding the CLZ. The implementation starts with 32 two-bit pairs: 00 has two leading zeroes, 01 has one, and a pair starting with 1 has none. Five merge layers combine these counts into the result for the full 64-bit input. In clz_merge, the highest bit indicates that a whole block is zero; the remaining bits encode its leading-zero count.

module clz_merge #(
    parameter WIDTH
) (
    input wire [WIDTH-1:0] left,
    input wire [WIDTH-1:0] right,

    output logic [WIDTH:0] out
);

    logic both_full, left_not_full;
    assign both_full = left[WIDTH-1] & right[WIDTH-1];
    assign left_not_full = left[WIDTH-1] == 1'b0;

    logic [WIDTH-2:0] right_chopped;
    assign right_chopped = right[WIDTH-2:0];

    logic [WIDTH:0] result;

    always_comb begin
        if(both_full) begin
            result = {1'b1, {WIDTH{1'b0}}};
        end else if(left_not_full) begin
            result = {1'b0, left};
        end else begin
            result = {2'b01, right_chopped};
        end
    end

    assign out = result;

endmodule

module clz_64 (
    input wire [63:0] data,
    output logic [6:0] out
);
    
    logic [1:0] bit_pair [31:0];
    logic [6:0] assembly_layer [5:0] [31:0];

    genvar j;
    generate
        for (j = 0; j < 32; j++) begin
            assign bit_pair[j] = data[2*j+1:2*j];

            assign assembly_layer[0][j] =
                (bit_pair[j] == 2'b00) ? 6'b000010 :
                (bit_pair[j] == 2'b01) ? 6'b000001 :
                                         6'b000000 ;
        end
    endgenerate

    genvar layer, i;
    generate
        for (layer = 1; layer < 6; layer++) begin
            for (i = 0; i < (32 >> (layer - 1)); i += 2) begin
                clz_merge #(
                    .WIDTH(layer + 1)
                ) m_clz_merge (
                    .left(assembly_layer[layer - 1][i + 1][layer:0]),
                    .right(assembly_layer[layer - 1][i][layer:0]),
                    .out(assembly_layer[layer][i >> 1][layer+1:0])
                );
            end
        end
    endgenerate

    assign out = assembly_layer[5][0];

endmodule

The squared components of the vector are shifted right by FP_SHIFT before being added, so $x$ still has FP_SHIFT fractional bits. If $F = \mathrm{FP\_SHIFT}$ and $X$ is the stored squared magnitude, its real value is $X/2^F$. The stored reciprocal-square-root estimate therefore needs the scale factor

The code uses e = (3*FP_SHIFT - b) >> 1 for this exponent. It zero-pads the squared magnitude to 64 bits before counting leading zeroes, then uses b = 63 - len_sqr_clz to find its most significant bit. The clocked assignment y_pipe[2] <= (1 << e) << FP_SHIFT gives the estimate an extra FP_SHIFT bits, which are removed by y_pipe_shifted before it is used in the next multiplication.

clz_64 m_clz_64 (
        .data({{(64-DOT_PRODUCT_WIDTH){1'b0}},len_sqr_pipe[1]}),
        .out(len_sqr_clz)
    ); // 1

    assign b = 63 - len_sqr_clz; // 1
    assign e = (3*FP_SHIFT - b) >> 1; // 1

Newton-Raphson Refinement

To improve this initial guess, I computed several iterations of a Newton-Raphson approximation scheme. Through a C simulation I found that 4 iterations was enough to achieve visually satisfactory results.

For an estimate $y_n$ of $1/\sqrt{x}$, the iteration is

At the desired value, $xy_n^2 = 1$, so the correction factor is one. Each iteration employs only multiplication and substraction operations. In fixed-point arithmetic, the products also need to be shifted to account for the fractional bits. I split each iteration across three clock cycles: square the current estimate, multiply by the squared magnitude, and apply the correction. THREE_HALFS stores 1.5 in fixed-point format. The expression half_ay2_pipe[i] >> (2 * FP_SHIFT + 1) accounts for the product’s extra fractional bits and divides it by two. The generated stages implement the equivalent form $y_{n+1} = y_n(1.5 - xy_n^2/2)$.

genvar pipe_idx;
    generate
        for(pipe_idx = 3; pipe_idx < 3 + 3*ITERATIONS; pipe_idx += 3) begin
            always_ff @( posedge clk ) begin
                y2_pipe[pipe_idx] <= y_pipe_shifted[pipe_idx-1] * y_pipe_shifted[pipe_idx-1]; // 3*i
                // y2_pipe[pipe_idx+1] <= y2_pipe[pipe_idx];
                // y2_pipe[pipe_idx+2] <= y2_pipe[pipe_idx+1];

                // half_ay2_pipe[pipe_idx] <= half_ay2_pipe[pipe_idx-1];
                half_ay2_pipe[pipe_idx+1] <= (len_sqr_pipe[pipe_idx] * y2_pipe[pipe_idx]); // 3*i + 1
                // half_ay2_pipe[pipe_idx+2] <= half_ay2_pipe[pipe_idx+1];

                y_pipe[pipe_idx] <= y_pipe[pipe_idx-1];
                y_pipe[pipe_idx+1] <= y_pipe[pipe_idx];
                y_pipe[pipe_idx+2] <= ($signed(y_pipe_shifted[pipe_idx+1]) * ($signed({1'b0, THREE_HALFS}) - $signed({1'b0, half_ay2_pipe_shifted[pipe_idx+1]}))); // 3*i + 2
            end
        end
    endgenerate

Precision and Pipeline Cost

In the evaluation described in our final report, using an 11-bit integer range and an FP_SHIFT of 12, the computed vectors were within 10% of the desired vectors, which was enough accuracy for our application. This is an observed result for the evaluated inputs rather than a guarantee for every fixed-point input; sufficiently small vectors can have their squared components truncated to zero.

When synthesized for an 11-bit integer range and 12 bits of precision, the normalizer consumes 48 DSP slices. The latency of the normalizer is $3 + 3I$, where $I$ is the number of iterations used for the inverse square root solver. While it is fully pipelined, all multiplications are assumed to take one cycle, which limits the range and precision of the input vectors due to the volume of operations required between frames. Too high a latency and we would not be able to finish computing updates for all boids.

Boid Computations

Every boid is associated with a position and velocity vector. Every frame, we add the velocity vector multiplied by a small $\Delta t$ to its position. The intended full behavioral model calculates the acceleration of the boid based on:

1. Normalized offset to the target’s position. Used in computing the target force.
2. Average direction of other boids in the perception radius. Also known as the average flock heading. Used in computing the alignment force.
3. Normalized offset to the average position of other boids in the perception radius. Also known as the flock center. Used in computing the cohesion force.
4. Average of the negative offset to the position of other boids. Also known as the separation heading. In essence, this force makes sure that boids don’t fly over each other. Used in computing the separation force.

In the full model, each of the above results is computed into a steering force by subtracting the boid’s velocity from the influencing vector. These results are then clamped to have a maximum magnitude equal to a constant maximum steering force. These contributions to the acceleration are then summed and added to the boid’s velocity.

The velocity’s magnitude is then clamped between a constant minimum and maximum speed. This new velocity is then used next frame to update the boid’s position.

Boid Update

The boid_update module computes the boid’s new velocity, which is passed back to the boid_manager module. In the submitted implementation, it only implemented the first of the four steering forces, which is responsible for leading the boid towards the target. To do this, it implements an FSM comprised of the following states:

1. IDLE
2. NORMALIZE_OFFSET
3. GET_STEER
4. NORMALIZE_STEER_TARGET
5. CLAMP_STEER
6. GET_ACCELERATION
7. GET_VELOCITY
8. NORMALIZE_VELOCITY
9. CLAMP_VELOCITY

Of relevance here is that, to clamp a vector between given minimum and maximum magnitudes, it is necessary to normalize it beforehand. Due to resource limitations, it was impossible to instantiate a separate normalizer module for each of these operations, so instead we opted for a blocking implementation during which the boid_update module changes the inputs to its normalizer submodule and waits for results.

To implement the rest of the steering forces, it would then be necessary to add stages for the computation of the necessary data as well as the necessary normalization stages before each steering force computation and subsequent clamping of the output value.

Our final report recorded a 51-cycle latency for boid_update and estimated at least 19,000 boid updates per frame on the pixel clock, including boid_manager overhead. In the supplied source version, the FSM takes 56 cycles after accepting an update request, before manager overhead: three 16-cycle normalization states, three two-cycle steering or clamping states, and two single-cycle states. The report’s throughput estimate therefore needs to be recalculated for this version.

Boid Manager

The boid_manager module responsible for passing the necessary parameters to boid_update for every boid implements an FSM comprised of the following states:

1. IDLE
2. POS_UPDATE
3. POS_UPDATE2
4. UPDATE_START
5. UPDATE_WAIT

During the IDLE state, the module awaits the new_frame signal before saving the currently inputted target coordinates and transitioning to the POS_UPDATE state. In this state, the module updates all boid positions by adding their respective velocities scaled by a small positive number. Then, during POS_UPDATE2, all positions are boundary-checked and wrapped around the screen if necessary.

During UPDATE_START, the manager asserts the data_in_valid signal to the boid_update module and then transitions to the waiting state, where the current boid index is incremented after the boid_update module’s busy signal goes low. This continues until all boids’ velocities have been updated, at which point the module returns to the IDLE state.

Design Evaluation

Double Frame Buffer and Rendering Bottleneck

The objective of the submitted implementation was to be able to render as many boids as possible and have them change sizes appropriately. Writing camera pixels into the frame buffer consumes most of the frame time, and the system cannot write an amount of boid pixels that surpasses the amount of cycles in the remaining frame time. This amount of pixels is roughly equal to the amount of cycles in the v-sync period.

This upper bounds the size and quantity of the boids by the amount of pixels for each boid when they’re closest to the screen. In the configuration evaluated in our final report, we found that we could comfortably render 16 boids as 96 × 96 squares — or 147,456 pixels — whilst remaining under the frame time constraint. This measurement differs from the supplied renderer’s current settings, which produce nominal square sizes of 4–49 pixels before eight-pixel width packing. This was the design’s bottleneck and would be the first area to be improved upon in the next iteration as both the depth calculation and boid updates can handle increased throughput.

Additionally, we could cut off the additional pixels sent via the eight-pixel packing by modifying the strobe signal to the DDR3 controller to only write pixels included in the square region. This would remove the current rendering artifact where box widths change only in increments of eight.

Communication Protocol and Depth Calculation

Since the communication protocol only needs to communicate 22 bits for the center of mass information every frame regardless of any changes to the rest of the design, it would not need any changes or improvements. Similarly, the depth calculation module deals with known constants and is mostly a wrapper for a fixed-point divide. Even if a different camera were to be used, it would only require constant values and/or bit widths to be changed in this module.

Vector Normalization and Boid Updates

The biggest source of resource allocation in the design comes from the normalizer instanced by the boid_update module. Forty-eight of the sixty DSP slices in the module come from the normalizer instance. Most of these come from the pipelined Newton-Raphson iterations, which instance multiplications between numbers with result bit widths of up to $4\,\mathrm{FP\_RANGE} + 3\,\mathrm{FP\_SHIFT} + 2$. This results in timing and resource utilization issues for higher integer ranges and precision widths.

This problem could be solved by splitting each multiplication into different steps — each of them fitting into a single DSP slice. This would drastically reduce resource utilization and improve throughput at the cost of increased latency. Increased latency, however, directly influences the amount of boids that can be updated every frame due to the boid manager’s blocking implementation of updates and the choice to instance only one normalizer for boid_update.

Project source (MIT GitHub)

Here’s a small overview of my favorite parts of OS.

Copy-On-Write

I implemented copy-on-write for fork. Instead of immediately copying a process’s memory, the parent and child initially share physical pages. Pages that were writable become read-only, and a write triggers a page fault that creates a private copy.

I added a page-table flag to distinguish these pages, reference counts to track their lifetime, and handling for writes made through the kernel’s copyout function.

NIC Driver

I implemented transmit and receive handling for the emulated E1000 Ethernet device, along with UDP port binding and packet queues. A process waiting for a packet sleeps until the receive path wakes it up.

The driver uses rings of descriptors shared with the OS via MMIO. I replace received buffers before handing packets to the network stack, and reclaim transmitted buffers once the device is done with them.

char *packet;
while((packet = (char *) udp_port->packet_ring[udp_port->head]) == 0) {
  sleep(udp_port, &netlock);
}

Letting CPUs Work in Parallel

I split the physical memory allocator into per-CPU free lists, each with its own lock. When one list is empty, it can take a page from another CPU’s list. I also implemented a reader-writer spinlock that allows multiple readers, with pending writers preventing new readers from entering.

Extending the File System

I added doubly indirect blocks so files could grow beyond xv6’s original limit, including the cleanup needed when truncating them. I also implemented symbolic links, following chains when opening a file and supporting O_NOFOLLOW.

Mapping Files into Memory

I added file-backed mmap and munmap. Each process records its mapped regions, and page faults load file contents as those pages are accessed. Shared mappings write data back during unmapping, while private mappings keep changes out of the file. I also added handling for these regions across fork, exit, and exec.

Accessing a file through a pointer still involves allocating pages, checking permissions, and managing the underlying file’s lifetime.

The primary goal of YGo was to achieve availability: it is disruptive for a user to be unable to create and edit documents because all resources are currently exhausted. We designed the signaling layer around near-stateless servers, allowing for quick horizontal scaling in case of need. By “near-stateless” we mean that some state is stored in each signaling server’s memory, but losing the peer registry does not itself erase the clients’ document state. Established peer connections can continue without signaling; connecting or reconnecting peers depends on recovering the room assignment.

A second goal was to achieve fault tolerance: if one of the signaling servers dies, clients should still be able to create documents and establish P2P connections, provided the backend retains a Raft quorum. Since the servers relay signaling traffic between clients, we designed the system so that every client within the same document connects to the same signaling server. Without carefully addressing the failure of a hosting server, there could be split brain in the client network, causing two or more groups of clients to be unable to communicate with each other.

Contributions

I built YGo with Carolina Moura, with whom I coauthored the final report. I worked on the client described in Section III: the YATA list and text CRDT, its interface, and the peer-to-peer synchronization protocol. The Raft and key-value server code also came from my 6.5840 labs. For this project, we adapted the storage system to use gRPC instead of the simulated RPC system provided for testing.

System Overview

A client can query any YGo node on its assigned HTTP port to create a document, referred to as a “room,” or look up its hosting signaling server. The client then opens a WebSocket to that host and receives the list of current collaborators. Via the WebSocket, clients exchange the information needed to establish WebRTC connections within the same document. Document edits travel over WebRTC data channels rather than through the Go server.

On startup, each backend node starts an HTTP server, a gRPC server, a key-value instance, and a Raft instance. Upon client requests, the HTTP server calls functions provided by the Clerk, which calls the KV service over the network. The replicated state machine under the KV server uses Raft as its consensus algorithm. Raft and KV share the gRPC listener and the client connections between nodes.

The distinction between document state and room metadata is central to the design. The replicated backend stores room assignments and signaling-server heartbeats. The document itself lives in the peers’ CRDT instances. Replicating metadata therefore does not make document content durable.

The YATA List and Text CRDT

We employed the YATA approach to a Conflict-free Replicated Data Type, whose basic primitive is the YATA list. Peers can operate concurrently on different versions of the list and share updates without requiring consensus amongst themselves. Convergence depends on integrating the same updates using deterministic ordering rules.

A list item stores its identifier, creation-time origins, mutable linked-list neighbors, content, and a deletion flag:

The origins refer to the IDs of the items immediately to the left and right of $i$ when it was first created. The left and right fields are mutable pointers used to maintain the local view of the linked list. A boolean flag, or tombstone, is used for deletions to preserve the structure: an item is marked deleted rather than removed, so concurrent insertions referencing its ID as an origin can still be resolved.

Each item is assigned an ID consisting of a pair:

where $f$ identifies the client and $c$ is a monotonically increasing insertion counter maintained by that client. The report described $f$ as a UUID. In the supplied implementation, the browser hashes the signaling UUID with SHA-256 and uses the first signed 32-bit word as its numeric CRDT identifier. This is an identifier scheme rather than a guarantee of uniqueness; collisions remain possible.

Deterministic Insertion

When integrating item $i$, the local list may already contain other insertions between originLeft and originRight, made after $i$ was created. To find its position deterministically, the algorithm traverses the local list from the left origin’s right neighbor toward the right origin.

For items sharing the same left origin, it breaks ties by client identifier. An item with a smaller client identifier is resolved to the left of the incoming item. If it has the same right origin and does not sort to the left, the scan stops. The algorithm also tracks items encountered during the scan and unresolved conflicting items, so insertions whose origins lie within that region retain their relationship to preceding items. An item with a left origin outside the scanned region ends the scan. The new item is then linked immediately after the rightmost item resolved as being to its left.

The report defined a counter-first total order over IDs, but the insertion code uses client identifiers for the same-origin tie-break. The excerpt below shows that implemented rule and the dependency tracking directly.

    while (!ID.equals(scan.id, right.id)) {
      itemsBeforeOrigin.add(ID.key(scan.id));
      conflictingItems.add(ID.key(scan.id));

      if (ID.equals(scan.originLeft, item.originLeft)) {
        // tie-break by UUID
        if (scan.id.userID < item.id.userID) {
          left = scan;
          conflictingItems.clear();
        } else if (ID.equals(scan.originRight, item.originRight)) {
          break;
        }
      } else if (itemsBeforeOrigin.has(ID.key(scan.originLeft))) {
        if (!conflictingItems.has(ID.key(scan.originLeft))) {
          left = scan;
          conflictingItems.clear();
        }
      } else {
        break;
      }

      scan = this.getItem(scan.right);
    }

When inserting multiple items at the same time, they may not be in dependency order. applyUpdate repeatedly integrates items whose origins are already known. If it drains the pending set, integration is complete; if a pass makes no progress and pending items remain, it throws an unresolved-origins error. For an item already present, it merges deletion state monotonically: an incoming tombstone can mark the existing item deleted, but cannot restore it.

We implemented a simple Text CRDT using the YATA list by making each insertion an individual character. Allowing arbitrary-length strings to be inserted as a single item, and splitting them only when required by later operations, remains a potential optimization.

State Vectors and Deletions

A state vector records how many insertions from each client the peer knows. In the source, the stored value is the next counter after the greatest integrated counter:

with zero used when none are known. makeUpdate sends items for which $c \geq SV_{\mathrm{recipient}}[f]$. This representation assumes that the counter summarizes a complete prefix; it does not encode arbitrary gaps in a client’s insertion history.

    const items: YataItem[] = [];
    for (const item of this.itemById.values()) {
      if (this.isSentinel(item)) continue;
      const known = target.get(item.id.userID) ?? 0;
      if (item.id.opCounter >= known) {
        items.push(item);
      }
    }

Deletions do not allocate new insertion IDs. The protocol therefore sends tombstone keys separately from the insertion diff. getItems returns a left-to-right ordered list of items that have not been deleted, while the underlying tombstoned items remain available for resolving origins.

Peer-to-Peer Synchronization

A group of peers sharing the same text instance communicates edits through TextPeer, a protocol layer that wraps the CRDT. The wire format has two message kinds: stateVector, carrying the sender’s full state vector, and update, carrying CRDT items and tombstone keys.

Initial Sync

When a peer’s data channel to another peer first opens, it calls syncWithPeer, which sends a state-vector message. The receiver replies with an update containing every item the sender is missing, computed with makeUpdate, plus every tombstoned key. Because the data channel opens on both sides, the exchange runs symmetrically so each peer can acquire the other’s state at the time of synchronization.

Periodic Broadcast

After initial sync, a timer in the application layer calls broadcastChanges. Each call builds an item diff against lastBroadcastSV and a deletion list from tombstones appended since the previous broadcast. If either list is nonempty, it sends the update to every connected peer and advances both broadcast cursors.

  broadcastChanges(rtc: RTC): void {
    const items = this.text.makeUpdate(this.lastBroadcastSV);
    const deletes = this.tombstoneOrder.slice(
      this.lastBroadcastTombstoneCount,
    );
    if (items.length === 0 && deletes.length === 0) return;
    const msg: WireMessage = { kind: "update", items, deletes };
    rtc.broadcast(JSON.stringify(msg));
    this.lastBroadcastSV = this.text.getStateVector();
    this.lastBroadcastTombstoneCount = this.tombstoneOrder.length;
    console.log("broadcastChanges", JSON.stringify(msg));
  }

handleMessage applies incoming items and tombstones. In the supplied source, it leaves the broadcast cursors unchanged, allowing items and tombstones learned from one neighbor to ride out on the next broadcast to other neighbors. The cursor-advancing optimization in that method is commented out because it would suppress forwarding in a partially connected graph.

A useful property of this design is that the graph of peers need not be fully connected for updates to propagate. The design uses forwarding to propagate updates along paths in a connected component, provided dependencies can be integrated and messages continue to be delivered. Disconnected components cannot exchange edits until a path is restored. The browser’s connection setup nevertheless attempts a mesh by connecting each new peer to the peers listed in its welcome message.

The supplied UI defaults to a two-second broadcast interval, adjustable from 200 ms to 10 seconds in 200 ms steps. These are batching settings, not measured end-to-end editing latencies. The local text changes immediately; remote visibility depends on the next broadcast, transport, and integration.

Why broadcasting?

It’s possible to implement the peer-to-peer protocol as a pull-based protocol instead of a push-based protocol. This would entail deciding when to request updates instead of when to send them to peers. I decided to stick with a push-based protocol for several reasons. First, it is much simpler to think about and implement. Broadcasting peers need not wait for a reply and receiving peers don’t have to send one. Second, it removes needless probing that would occur with a pull-based approach. Peers can be optimized to only broadcast when they have something to share, reducing traffic. A notification-push and pull-based approach could circumvent these kinds of issues, but there is very little gain in complicating the approach in this way.

Signaling and Room Assignment

The signaling server fulfills three purposes: it manages the creation of rooms, answers requests about which server currently hosts a room, and acts as a middleman in establishing P2P connections if it is the hosting server.

The storage interface provides versioned reads and conditional writes:

GET(k) → (value, version)
PUT(k, v, version) → ok | ErrVersion | ErrMaybe

A successful write increments the version. ErrVersion indicates a conflicting version; ErrMaybe represents an uncertain outcome after a retry. This distinction matters because a failed response does not necessarily mean the original write was not applied.

The report described separate server/<addr> lease keys. The supplied source instead stores a serialized address-to-timestamp map under one heartbeats key. Each server updates its entry with a versioned read-modify-write. Room ownership is stored separately as room/<id> → advertised address.

Heartbeats and Room Creation

Each server registers its advertised address and periodically renews its timestamp. With the supplied startup configuration, renewals are scheduled every 10 seconds and Signalings uses a 30-second freshness window. A server belongs to the live set when

These values describe the implementation’s timers; they are not a bound on failover time. Divergent wall clocks, failed renewals, retries, and delayed requests all affect detection.

On POST /api/rooms, the server lists the live signaling set, picks a host uniformly at random, generates a fresh six-character room ID, and attempts a version-zero write. The report called these IDs alphanumeric; the current generator uses uppercase letters only. The receiving server need not be the chosen host: any signaling server can create a room on behalf of another. The current creation loop generates a new ID after any non-OK result, rather than distinguishing only version conflicts.

Lookup and Reassignment

On GET /api/rooms/:room, the server reads the room assignment and checks whether the recorded host is still in the live set. If it is, its address is returned. If its heartbeat has expired, the server picks a new host and writes it with the version observed during lookup. If another lookup raced this update, ErrVersion causes the check to repeat.

		// The server is dead, pick a new random server
		newServer := live[rand.Intn(len(live))]
		putErr := s.ck.Put("room/"+room, newServer, version)

		if putErr == rpc.OK {
			fmt.Printf("Room %q reassigned: %q (dead) -> %q\n", room, server, newServer)
			return newServer, rpc.OK
		}
		if putErr == rpc.ErrVersion { // someone raced us; prob will get out in the next iteration
			continue
		}
		return "", putErr

The host can die between the lookup and the client’s connection attempt. The client discovers this when connecting fails and repeats the lookup. In the supplied frontend, reconnection starts with a 500 ms backoff, doubles on repeated failures, and caps at 10 seconds. Lookup tries the page’s origin and a hard-coded list of three localhost servers, which would need to be replaced for deployment.

Establishing Peer Connections

Before upgrading a WebSocket, the server validates the room and checks that it is still the assigned host. On a successful upgrade, it assigns a fresh peer UUID, inserts the connection into its in-memory room registry, and sends a welcome message carrying that ID and the IDs of current peers.

Peers use signaling to exchange SDP offers, answers, and ICE candidates. The report described all peer JSON as broadcast; the source stamps the sender’s ID and unicasts messages with a to field to that peer. Messages without a target are broadcast to the other peers in the room. Once WebRTC data channels are established, edits use those channels.

Recovering from Signaling Failure

We addressed two kinds of failure. When the hosting signaling server dies, its heartbeat eventually expires. A subsequent lookup through a surviving node reassigns the room. Clients notice the WebSocket close and repeat the lookup, reconnecting to the new host. The frontend keeps its RTC instance and CRDT across the signaling outage, so the outage alone does not tear down established data channels or discard document state. Recovery behavior also depends on subsequent peer-membership messages and reconnection.

A server can also be reassigned away from a room while still alive if it misses enough heartbeats. It may still hold open WebSockets for rooms it no longer owns. To address this stale registry, each server runs an eviction check every 10 seconds. If it discovers it is no longer the assigned host, it closes the room’s WebSockets, triggering client lookup and reconnection. This is a periodic recovery mechanism, not an instantaneous fencing guarantee.

Replicated Key-Value Storage

For the storage system, we used the code from Lab 4 and adapted it to gRPC. We also used the lab’s Raft code beneath the replicated state machine that sustains the KV server. Both Get and Put handlers submit operations through the RSM; the Clerk retries against other nodes when it receives ErrWrongLeader or cannot reach a server.

The KV state machine accepts a write only when its expected version matches the current value. For a new key, the expected version must be zero.

	vv, ok := kv.store[args.Key]
	if !ok && args.Version != 0 {
		reply.Err = rpc.ErrNoKey
		return
	}
	if ok && vv.Version != args.Version {
		reply.Err = rpc.ErrVersion
		return
	}
	kv.store[args.Key] = VersionedValue{Value: args.Value, Version: args.Version + 1}
	reply.Err = rpc.OK

The source uses a bbolt-backed persister for Raft state and snapshot bytes. This storage supports the replicated metadata layer; it does not store the browser’s text CRDT. KV snapshot and restore methods exist, while bootRaft passes -1 as the RSM’s snapshot threshold, disabling automatic size-triggered snapshots in this configuration.

The report assumed an always-available, strongly consistent storage service when explaining signaling. In practice, committing these operations requires a Raft majority. The supplied Compose configuration contains three nodes, so two are required for a quorum. Adding signaling capacity does not itself provide automatic Raft membership changes. Existing peer editing can continue independently of the backend, but room creation, lookup, and reassignment depend on the metadata service making progress.

Results and Limitations

We implemented the browser-side YATA list and character-based text wrapper, initial state exchange, periodic edit broadcasts, WebRTC signaling, versioned room assignment, heartbeat-based discovery, and periodic eviction of stale room registries. The final report described the architecture and recovery paths but did not include quantitative latency, throughput, scalability, or failover measurements. The timer values above are source configuration, and the recovery paths describe mechanisms rather than experimentally established guarantees.

The project has several limitations:

• Document durability. Content is not saved to persistent storage. Once the last peer leaves a room, the document content is lost even if its room assignment remains in the database.
• Resource exhaustion. There is no authentication or rate limiting to prevent a user from creating large numbers of rooms. Room records accumulate in the metadata store; retained items and tombstones also grow with edit history.
• Wall-clock leases. Heartbeat freshness relies on each machine’s clock. Clock differences can cause a node to be declared dead prematurely, and storing all heartbeats in one map introduces contention on that key.
• Editor scope. The supplied UI handles single-character insertion, line breaks, and single-character deletion. It rejects range edits, paste, IME composition, and other multi-character inputs.
• Update handling. Unknown tombstone keys are ignored, and unresolved insertion origins cause an exception rather than entering a persistent retry queue. These paths limit the implementation’s tolerance of missing dependencies; connected topology alone is not an unconditional delivery or convergence guarantee.

Planned work included a Sync operation to store CRDT state in the database, client snapshots for document persistence, authentication, rate limiting, a load balancer, and a more reliable lease implementation. Arbitrary-length text items remain another potential optimization. These features are not implemented in the supplied source.

Source Code and References

Project source.

The underlying Raft and KV code originated in my course labs. The report asks readers to obtain permission before accessing that lab repository.

The CRDT design followed P. Nicolaescu, K. Jahns, M. Derntl, and R. Klamma, Near real-time peer-to-peer shared editing on extensible data types, GROUP ‘16, pp. 39–49 (DOI).