Go 1.27 and the "minor library change" to compress/flate

§

WebSockets have been around quite a while now, and I've been using them in Go services. There is an extension in the spec for message compression, called permessage-deflate. You negotiate it over Sec-WebSocket-Extensions: permessage-deflate, and in theory it's simple: compress each message with DEFLATE, send it, let the peer decompress. In practice it's a silent contract between two libraries: a compression library (Go's compress/flate) and a websocket library (gobwas/ws).

💡 Pay attention to the fact that these are two separate RFCs. DEFLATE is RFC-1951, and permessage-deflate is RFC-7692 which utilizes RFC-1951. This will come back up later.

When an implementation of a foundational standard changes (even in an entirely valid way), it can ripple upward to implementations of standards based on said foundational standard's original implementation. 🤯 That's exactly what a Go 1.27 "minor library change" did recently; let me show you the byte that bit me (and a few others).

Disclaimer

Until this happened, I never had reason to look behind the curtain at permessage-deflate stuff. It's really "just worked" (except iOS and Safari clients 😠), for which I'm grateful! Having to dig a bit deeper to resolve an issue for my team, I took the opportunity to read up on it have AI explain it to me like I'm 5 (and, I also read up on it). This is how I understand things to work, so if something's off it's not intentional.

Also, there is no accusation here that Go did anything wrong (they didn't). I love Go, have used it for years, and think it's just fantastic 😍

The permessage-deflate Tail Contract

RFC 7692 §7.2.1 of the permessage-deflate spec discusses how a compressed message must end. But to get there, we have to consider the whole stream.

A DEFLATE stream (foundational spec) is a series of blocks, and each block starts with a header byte carrying two things:

  • BFINAL 0|1 is this the final block?
  • BTYPE 00|01|10 how the block is encoded

Quick primer on the three block encoding types, because the name "DEFLATE" caused me some confusion here...

  • STORED (00): The block's bytes ship through as-is behind a tiny LEN/NLEN header. No Huffman codes, no back-references; raw data in a deflate envelope. Compressors fall back to it when data won't compress, or when the Huffman table would cost more bytes than the payload itself.
  • fixed-Huffman (01): Symbols are Huffman-coded against a table that's defined in RFC 1951 and known to every inflater, so the sender transmits no table. Quick, but not tailored to your data.
  • dynamic-Huffman (10): The encoder computes a per-message Huffman tree and transmits it (as code lengths) ahead of the payload. Best compression, but the table itself costs bytes.

The spec requires the final block of every compressed message to be an empty STORED block, i.e. BFINAL=1, BTYPE=00, with length zero:

0x01     |  0x00 0x00  |  0xFF 0xFF  
BFINAL+  |  LEN (2B)   |  NLEN (2B, one's-complement of LEN)  
BTYPE    |  = 0        |  = 0xFFFF

That makes the final block a 5-byte sequence 01 00 00 FF FF. Here's the subtle bit: the sender must strip the last 4 bytes (the LEN/NLEN pair) before the bytes reach the wire, and the receiver re-appends 00 00 FF FF before inflating.

Since the final block is empty, stripping LEN/NLEN is zero-cost (there's no data to lose). So in practice a compressed payload travels with a single 01 byte on the end, and the peer rebuilds00 00 FF FF tail before inflating. Which means: a correctly terminated permessage-deflate payload ends with 0x01 on the wire.

💡 Emphasizing the wire here is important. We're still talking about the permessage-deflate spec, not the foundational DEFLATE spec.

Go 1.26 and Earlier

Up through Go 1.26, flate.Writer.Close() wrote exactly that final empty STORED block:

Close()   =>  compressed-data ... 01 00 00 FF FF
wire      =>  compressed-data ... 01               // LEN/NLEN stripped
receiver  =>  compressed-data ... 01 00 00 FF FF   // LEN/NLEN re-added      
          =>  inflate through end-of-stream

gobwas/ws (specifically its wsflate package) is built to expect that implementation. It keeps the last 4 bytes it sees in a sliding window and, on close, runs checkTail():

func (w *Writer) checkTail() {
  if w.err == nil && w.cbuf.buf != compressionTail {   
    // compressionTail == [4]byte{0,0,0xff,0xff}
    w.err = fmt.Errorf(
      "wsflate: bad compressor: unexpected stream tail: %#x vs %#x", 
      w.cbuf.buf, 
      compressionTail, 
    )
  }
}

In other words: "the last thing I saw must be 00 00 FF FF (the LEN/NLEN pair of the final STORED block) so I can omit flushing them to the wire". If the check fails, the message is refused.

What Go 1.27 Changed

Go 1.27 shipped a rewrite of compress/flate. Among the compression improvements (it does compress a bit better now! 😉), it changed one small detail: the final block written by Close() is no longer an empty STORED block. It's now an empty fixed-Huffman block costing only ~10 bits (2 bytes) with no LEN/NLEN or stored-block envelope; Go's source even comments

To write EOF, use a fixed encoding block. 10 bits instead of 5 bytes.

A STORED block always ships that 5-byte (40 bits) 01 00 00 FF FF:

Close()   =>  compressed-data ... 0x03        // 10-bit fixed-Huffman EOF, header byte 0x03
wire      =>  compressed-data ...             // wsflate doesn't see the "tail" it expects...

0x03 doesn't end with 00 00 FF FF, so checkTail fires and we get this error:
wsflate: bad compressor: unexpected stream tail: 0x03800100 vs 0x0000ffff

Now, this is where the distinction of the foundational spec comes into light. 0x03 is a perfectly valid final deflate block. A fixed-Huffman block with an empty code sequence and BFINAL=1. Standalone compress/flate users never notice the difference. It's an assumption mismatch, not a bug: wsflate assumes the stream ends the RFC 7692 way, and Go 1.27 stopped writing the bytes that spec asks for. (See gobwas/ws issue#221)

It was only by coincidence that up to this point, the implementation for DEFLATE (again, not to be confused with permessage-deflate), happened to add the extra bytes at the end. Maybe this was a carry-over from some prior convention (AI says zlib does it this way), but RFC 1951 just defines blocks and doesn't mandate the terminal block's shape. My guess: a maintainer saw that the DEFLATE spec allows a more condensed terminus and implemented it.

Understanding the two separate specs in more depth (again, not an expert) was the key for me: compress/flate in Go's std lib is not intending to implement permessage-deflate spec at all! Its terminal bytes just happened to align for a long time. It's intending to implement the DEFLATE spec, which is used by any number of consumers (gzip, png, pdf, etc...).

Interestingly enough, Go's maintainers updated the docs in this commit, "compress/flate: clarify compatibility promise", adding an explicit note to the NewWriter docs (and the gzip, zlib, zip, and PNG writers that wrap it) that the exact bytes written are not covered by the Go 1 compatibility promise. Seems fair to me - the output bits are an implementation detail. But permessage-deflate is the one place where that "implementation detail" crosses a protocol boundary: a peer's inflator, and a websocket library's tail check, are both depending on bytes the Go docs now explicitly flag as unspecified.

Three Potential Outcomes

One option, and the quick fix I started with when I first hit this, is "don't call Close(); just call Flush()". It's a simple workaround, and it almost works. So let's look at what each option actually puts on the wire. You might be surprised at the difference a single bit makes 😅.

Option A: Close() (changed on 1.27)

As shown above: emits 0x03, checkTail fails, messages error out.

Option B: Flush() only (my first workaround)

Flush() emits 00 00 00 FF FF: an empty STORED block with BFINAL=0 😬 (indicating the stream is not final, just "we're at a flush boundary"). wsflate's cbuf holds the trailing 00 00 FF FF in its window (that's what checkTail validates and gets stripped from the wire), so the payload actually ends with just the 00 header byte:

Flush()   =>  compressed-data ... 00 00 00 FF FF   // BFINAL=0
wire      =>  compressed-data ... 00               // LEN/NLEN held by cbuf
receiver  =>  appends its own terminator anyway, inflates OK (probably)

checkTail passes, because the last 4 bytes it sees are 00 00 FF FF. And a gobwas/ws reader works, because wsflate.Reader defensively appends its own tail before inflating. But a strict zlib/RFC reader will see a stream that ends with BFINAL=0 - which means "more blocks follow" - and hit Z_BUF_ERROR the moment the stream just... ends. It works against the one library we're testing against, but it isn't fully RFC 7692-compliant. Flush() is a workaround, not the contract.

Option C: Flush() + append the terminator (fully compliant)

Do the flush (that gets byte alignment and satisfies checkTail's window), then append the RFC 7692 final block ourselves - through wsflate's cbuf, not the destination writer! Traced through the cbuf's 4-byte window: the flush emits a stored block whose LEN/NLEN lands in the window; appending the terminator pushes that 00 00 FF FF out onto the wire and parks the terminator's own LEN/NLEN in the window (which is what checkTail validates and strips):

data      =>  compressed-data ... 03 00            // last data bytes
Flush()   =>  compressed-data ... 03 00 00         // header byte out; 00 00 FF FF in window
append    =>  compressed-data ... 03 00 00 00 00 FF FF 01   // window flushed, 01 out
receiver  =>  ... 01 00 00 FF FF                   // re-adds LEN/NLEN
          =>  inflate -> clean Z_STREAM_END

The wire ends with 0x01, exactly the contract from the top of this post. Every receiver, strict or lenient, sees a proper end-of-stream.

The Fix in Practice

The way I'm using compression is that I only compress if the messages are going to be > 1k. Under 1k, I don't compress. As such, I found it easier in the first place to have a facade over the flate.Writer. So I was able to just implement a custom Close on my compressed writer (as it just implements io.Closer). So whether or not a message is being compressed, my socket connection just works with a singular set of interfaces:

var w io.Writer = connection.writervar 
fw io.WriteCloser // fw is just so I recognize it's really a flate.Writer
if shouldCompress {    
  // this would use my facade compressed writer, and ultimately writes to connection.writer    
  // this also gets a writer from a pool, and resets it for this new use (implementation detail)    
  fw = getFlateWriter(...)     
  w = fw // our Writer and our WriteCloser are same entity
}

w.Write(msg) // send the msg to the Writer (it will either compress or it won't... not the concern here)

if fw != nil {    
  // if we have a fw, we need to close it b/c we use a writer pool and this returns it to the pool - doesn't actually close the underlying writer    
  // but this is also where we have the opportunity to fix up the tail bytes!    
  fw.Close()
}

// finally, flush the main writer 
// by this point it's has had everything written to it that we want (compression or not)
connection.writer.Flush()

Reproduction

If you'd like to see all of this for yourself, I wrote up a quick example repo that shows each of these scenarios and test cases

-Bradley

ASTs in Go - Part 1 Previous Post