An 11-byte malicious input can force OpenSSL servers to allocate large amounts of memory and remain permanently bloated, Okta's Red Team reports.
The HollowByte mechanism in OpenSSL's TLS handshake
Okta's Red Team lays out a simple but effective denial-of-service technique it calls HollowByte. The issue arises during a TLS handshake: each message begins with a 4‑byte header that contains a three‑byte length field declaring the size of the handshake data that should follow. Vulnerable OpenSSL versions allocate the claimed length before the server actually receives and validates the payload.
An unauthenticated attacker therefore can open a TLS connection and send an 11‑byte input whose header declares a much larger message body. Because the server trusts the header and allocates the indicated memory, the worker thread then blocks "waiting indefinitely for data that will never arrive," Okta explains. Repeating this across many connections lets an attacker provoke large cumulative allocations from a small transmitted volume.
glibc allocator behavior and why memory stays bloated
Okta details why freed allocations do not immediately undo the damage on many Linux hosts. While OpenSSL frees the buffers when a connection drops, the GNU C Library (glibc) has an allocator behavior that "does not immediately return small‑to‑medium allocations to the operating system; it keeps them for potential reuse." By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks, Okta says, and "the heap fragments heavily, causing the server’s Resident Set Size (RSS) to climb continuously. Even after the attacker disconnects, the server remains permanently bloated." The only full remedy, according to the advisory, is to restart the process.
Real‑world effects shown in Okta's NGINX tests
Okta tested HollowByte against NGINX and found that low‑capacity environments can be easily depleted of memory with the technique. Higher‑spec servers may still lose a significant fraction of available memory — Okta reports figures of up to 25% loss in their tests — while the attack bandwidth remains below security alerting thresholds. The advisory notes that although DoS flaws are often considered less severe than those enabling data theft or code execution, they can still cause operational disruption and reputational damage.
OpenSSL releases that fix HollowByte and the recommended action
The OpenSSL project has applied a silent fix (no identifier assigned) and backported it to older releases. The changes are present in OpenSSL 4.0.1 and have been backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. These fixed builds "grow the buffer only when the data arrives, ignoring header claims," removing the root cause of the excessive preallocation.
Okta frames the change as a "hardening fix" rather than a labeled security vulnerability, but still recommends administrators "upgrade your distribution's OpenSSL packages immediately." The advisory underscores the breadth of exposure: OpenSSL is embedded in widely used web servers (NGINX, Apache), language runtimes (Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL), and it is commonly preinstalled on Linux distributions for TLS and certificate handling.
What technologists, enterprises, and distribution maintainers will watch
- Technologists and security teams: prioritize identifying hosts running the affected OpenSSL releases and schedule upgrades to OpenSSL 4.0.1 or the backported 3.x releases. Monitor process RSS for unexplained growth and be prepared to restart affected processes where upgrades cannot be applied immediately.
- Enterprises and procurement leaders: audit externally facing TLS endpoints and bundled software (web servers, language runtimes, databases) that ship with or rely on system OpenSSL libraries; plan for patch windows since the issue can cause operational outages even if no data breach occurs.
- Distribution maintainers and package managers: coordinate distribution updates and ensure packages are rebuilt and delivered; Okta's recommendation to "upgrade your distribution's OpenSSL packages immediately" places a premium on timely backports and clear guidance to downstream users.
Okta's advisory and the OpenSSL fixes close a deceptively small attack vector: an 11‑byte packet can compel a server to reserve memory at scale and linger in a bloated state until the process restarts. For environments that cannot immediately apply patched OpenSSL binaries, monitoring for unusual RSS increases and preparing safe restart procedures will limit operational exposure while distributors roll out fixes.
Original reporting: https://www.bleepingcomputer.com/news/security/hollowbyte-ddos-flaw-bloats-openssl-server-memory-with-11-byte-payload/
