A modern X11 server for macOS, built in 30 days by a seasoned programmer and Agentic AI

Smooth at device scale

A 1-bit window mask at X logical resolution renders blocky stairsteps on Retina. macXserver converts the mask to a CGPath and lets Core Graphics rasterize it at device scale. xeyes and oclock look round.

May 28, 2026

On Day 24 we shipped the SHAPE extension. That meant oclock finally rendered as an actual circle instead of a square clock face in a square window, and xeyes rendered as a pair of eyes in an oval window instead of a rectangle. That was the goal. The visual win you can show in a screenshot.

But when I looked at the result on my Retina display, the edges of those round shapes looked off. Not subtle. The oval of xeyes had visible blocky stairsteps every few pixels around the curve. oclock’s clock face had the same kind of jaggies on its circular outline. The shapes were right at a logical level (the curves were going where they should), but they didn’t look anywhere near the iTerm2-quality bar I’d set for the project.

This was right after the Pixel perfect work where Claude and I locked in integer-scale-only and AA-off-on-shape-fills as load-bearing rules. So I asked:

claude, why do the round shape windows look so blocky around the edges?

Same coordinate-system mismatch we hit before, just one layer up. Let me walk through what’s actually happening with the mask.

When a client like xeyes uses the SHAPE extension, it calls XShapeCombineMask with a 1-bit pixmap that defines which pixels of the window are part of the shape and which are transparent. That pixmap is in X logical pixels, the same coordinate space xeyes thinks in. xeyes builds the oval mask by drawing into a 1-bit pixmap that’s the same logical size as its window. Say 200×120 logical pixels.

At 3× scale on a Retina display, our actual NSWindow is 600×360 device pixels. If we apply that 200×120 mask literally, mapping each X logical pixel to a 3×3 device-pixel block, every edge bit of the mask becomes a 3×3 jagged step in the rendered window outline. The oval’s smooth mathematical curve gets quantized first to the source bitmap’s 1-pixel resolution, then quantized again by the 3× upsampling. On a Retina display where individual device pixels are tiny, those 3-device-pixel terraces look nine times bigger than they should. The shape is technically correct at the logical level. Visually it’s 1990s pixel art.

This is the same lesson we hit with the Motif XmText caret on Day 18 . That bug was: the caret pixmap was allocated at logical resolution, so when the caret got blitted into the window’s device-scale backing, the edges of the I-beam stipple were pre-quantized to logical pixels and lost all crispness. The fix there was to allocate pixmap storage at the window’s device scale, not its logical scale. Same general principle as here: anything that’s going to affect per-pixel rendering has to live at the same resolution as the pixels. Window masks. Pixmaps. Clip regions. Cursors. Everything.

The fix for the SHAPE mask is to convert the 1-bit pixmap to a CGPath before applying it:

  1. Trace the boundary of the masked region in the 1-bit pixmap. The tracing uses a marching-squares-style walk over the edge bits and emits a polygon. For an oval like xeyes’, that’s a few hundred line segments at integer logical coordinates.
  2. The polygon is now a vector representation, not a bitmap. The shape is still quantized to the source pixmap’s resolution, but it’s described as edges rather than pixels.
  3. Set the NSWindow’s opaque region to that path. We use a CAShapeLayer with the path set as its path property, plus a clear background outside.
  4. Core Graphics rasterizes the path at the layer’s device backing scale, with anti-aliasing on. The polygonal edges get smoothed at device resolution, so the 3-device-pixel logical-pixel terraces become subpixel transitions instead.

This is the place where it’s OK to have anti-aliasing on, by the way. We turned AA off on X11 shape draws (PolyFillRectangle, PolyLine, etc.) because of the draw-to-erase semantics that xclock and friends rely on, described in Pixel perfect . But the window mask isn’t an X11 shape draw. It’s a Core Animation property that controls window-level compositing. CG rasterizes it once when the path changes, and AppKit composites the result. There’s no draw-to-erase pattern for window masks, so AA is just doing its normal smoothing job. On Retina, where individual device pixels are small, AA turns the polygonal edges into transitions that read as smooth curves.

So the practical result for xeyes: the oval mask comes in as 200×120 logical bits, gets traced to a CGPath with about 400 vertices, gets set as the window’s CAShapeLayer.path, and gets rasterized at 600×360 device pixels with AA. The edge transitions land within single device pixels instead of single 3×3 logical blocks. xeyes looks like xeyes.

The principle keeps coming back across this project in different forms. The +0.5 user-pixel offset was the same idea for stroke positioning: make sure the stroke decision happens at the right coordinate granularity. Cell-follows-font was the same idea for glyph cells: instantiate the font at integer pointSize so Core Text’s hinter works at device scale. Here it’s the same idea for window masks. Every time we try to do “logical pixels first, scale up to device later” instead of getting to device pixels directly, we get artifacts that look bad on Retina.

The Retina display turns out to be brutally honest about coordinate-system mismatches. At 1× everything looks roughly right because there’s not enough pixel density to show the seams. At 3× every half-resolution decision is visible. That’s also why we got the visual quality we did once we fixed these: a Retina display rewards getting the math right.