• Suyash Dongre

Introduction

The Ceph RADOS Gateway (RGW) provides an Amazon S3–compatible RESTful interface for object storage, allowing clients and applications to interact with Ceph clusters using familiar S3 APIs. While this compatibility greatly enhances usability and integration, it also exposes a large and complex attack surface typical of REST-based storage services. Ensuring the robustness and security of the RGW’s S3 interface is therefore essential.

In this project, we aim to evaluate the security and resilience of the RGW by applying stateful REST API fuzzing techniques. Specifically, we explore the use of RESTler, a powerful REST API fuzzing framework developed by Microsoft Research. RESTler automatically analyzes an API specification—such as an OpenAPI or Swagger definition—to infer request dependencies, generate test sequences, and systematically explore potential failure scenarios.

Project

In this section, we describe how RESTler was integrated into the Ceph project, how S3 authentication support was added, and the concrete bugs and issues discovered during fuzzing of RADOS Gateway (RGW).

Integration of RESTler into Ceph

• A pull request was submitted to integrate RESTler into the Ceph repository: #63566. This PR lays the groundwork for bundling or orchestrating RESTler-based fuzzing as part of Ceph’s testing infrastructure.
• The integration effort includes scaffolding to drive RESTler against RGW endpoints, wiring up configuration, and enabling the fuzzing infrastructure within the Ceph development environment.

Enabling S3 Authentication + Proxy Logic in RESTler

• To make RESTler viable for S3-style APIs, it was necessary to add support for S3 header signing (authentication) and proxying logic. This was done via a pull request to the RESTler upstream: restler-fuzzer#984.
• The PR extends RESTler’s engine to allow injection of custom signing hooks, enabling it to generate valid AWS-style signed requests that RGW will accept.
• In addition, proxy server logic was added (or extended) so RESTler can mediate requests (for example, signing through a proxy layer) or intercept/fix parts of requests before forwarding them.
• These enhancements make it feasible to use RESTler with real S3 authentication, which is essential for meaningful stateful fuzzing of the RGW.

Bugs & Issues Discovered via RESTler Fuzzing of RGW

Using the above integrated setup, the fuzzing campaign uncovered real defects in RADOS Gateway. Two especially notable issues:

• **Issue 72669:** RESTler discovered a crash in RGW caused by improper handling of the HTTP_X_AMZ_COPY_SOURCE header. When this header was present but empty (""), the code only checked that the pointer was not NULL—it didn’t verify that the string contained data.

  As a result, RGWCopyObj::parse_copy_location() attempted to access name_str0 on an empty string, triggering a crash. The fix involved adding proper validation to ensure the header is non-empty before parsing, preventing invalid memory access and improving input robustness.

• Issue 72502: This bug was discovered when uploading an object with an excessively long file name, RGW correctly identifies the error (File name too long) but returns an HTTP 500 Internal Server Error instead of the expected HTTP 400 Bad Request, as specified in the AWS S3 Error Response documentatio