Uncontrolled Recursion in HDF5 H5B__iterate_helper() via Crafted File

HDF5 through 2.2.0 contains an uncontrolled recursion vulnerability in H5B__iterate_helper() in src/H5B.c. When iterating a version 1 B-tree, the function does not verify that a child node’s level matches the expected level. That check is performed only in H5B__cache_deserialize() when a node is first loaded into the metadata cache. A crafted HDF5 file whose B-tree child pointer refers back to a node already in the cache, such as a node that points to itself, bypasses the check and causes unbounded recursion, leading to stack exhaustion and a crash. This can be triggered by calling H5Gget_info() or H5Gget_num_objs() on a group in the crafted file, which reaches H5B_iterate() through H5G__obj_info() and H5G__stab_count(). An attacker who can get a user or application to open a malicious file can cause a denial of service.