“HollowByte” flaw targets OpenSSL servers

A security flaw that could facilitate a denial-of-service (DoS) attack has been identified in the OpenSSL cryptographic software library. The flaw, dubbed “HollowByte”, enables an unauthenticated attacker to cause a server to allocate a disproportionate amount of random-access memory (RAM) by sending a specially crafted TLS request containing only 11 bytes.

The security flaw is associated with the mechanism used by earlier versions of OpenSSL to process incoming data during the establishment of a TLS connection. The TLS connection process begins with a “ClientHello” message sent by the client. Each TLS handshake message is preceded by a four-byte header that specifies the volume of data expected to be received. Three bytes within this header define the anticipated length of the message payload.

In vulnerable versions of OpenSSL, a memory buffer is allocated according to the size specified in the header before the actual volume of incoming data is received and validated. By sending an 11-byte request that declares a large payload but does not include the corresponding data, an attacker can cause the server to allocate up to 131 kilobytes of memory for each connection. Consequently, the server’s worker process remains blocked while waiting for data that is never transmitted.

Repeated exploitation of the flaw through numerous concurrent connections can gradually exhaust the server’s memory resources, ultimately disrupting the availability of the affected services. The attack does not require access to a user account, pre-existing access to the targeted system or the use of an additional exploit chain.

Although OpenSSL releases the allocated memory buffer after the connection is terminated, the GNU C Library (“glibc”) may not immediately return small and medium-sized memory regions to the operating system. Sending multiple requests that specify different payload sizes can cause memory fragmentation and a continuous increase in the Resident Set Size (RSS), which represents the amount of physical memory actively occupied by the server process.

Under such circumstances, terminating the connections may not be sufficient to fully recover the allocated memory. The server process or the affected service may therefore need to be restarted to restore the availability of the memory resources.

The flaw has a potentially broad impact because the OpenSSL library is used by a wide range of software products, including the Apache and NGINX web servers, the Node.js, Python, Ruby and PHP programming environments, as well as database systems such as MySQL and PostgreSQL.

As a result, internet-facing servers that rely on vulnerable versions of OpenSSL to process TLS connections may become potential targets of denial-of-service attacks designed to exhaust available memory resources.