The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization

Thank you for the thoughtful and substantive feedback. Below, we respond to each concern in turn and clarify both the motivation and the scope of our work.

(1) On the motivation and real-world relevance of audio-only geo-localization.
We agree that many real-world systems can leverage richer multimodal signals (e.g., video, metadata, timestamps). However, our focus on audio-only geo-localization is motivated by settings where such auxiliary information is unavailable, unreliable, or intentionally absent. A concrete and practically important example arises in public safety and emergency response. In many regions, local authorities routinely receive a large number of anonymous or spoofed phone calls (e.g., bomb threats or false alarms), where only raw audio is available. In some large metropolitan areas, such calls can occur multiple times per day. The ability to infer coarse-grained geographic plausibility from audio alone can help prioritize responses and filter clearly inconsistent or malicious reports, yielding tangible safety value. More broadly, spoken communication remains a dominant medium in everyday life (phone calls, voice messages, emergency hotlines), making audio-only inference a relevant and underexplored capability. Our goal is therefore not to replace multimodal systems, but to characterize and benchmark what is possible and not possible when audio is the sole signal.

(2) On potential sampling bias in the dataset construction.
The reviewer is correct that Aporee, as a user-uploaded platform, does not represent a random sample of all acoustic environments. We took several steps to mitigate this bias. First, during curation, we explicitly enforced a balance between clips with and without spoken language (50/50), reducing over-reliance on linguistic cues. Second, in the human filtering stage, annotators were instructed to preferentially retain clips reflecting everyday environments (e.g., streets, transportation, residential surroundings) rather than only iconic landmarks or touristic recordings. Importantly, while we initially collected over 30,000 candidate clips, the final ~1.4K samples represent a deliberately high-quality subset that meets strict criteria for audio clarity, geographic validity, and localizability. In this sense, the benchmark is not intended to mirror the raw Aporee distribution, but to isolate a controlled set of audio-location pairs suitable for studying model behavior.

(3) On the dataset scale and the “global” claim.
We acknowledge that the absolute scale of the benchmark is modest relative to large vision or language datasets, and that geographic imbalance remains. Our use of the term “global” is intended to indicate geographic coverage (72 countries/regions across multiple continents), rather than statistical representativeness of all environments worldwide. We agree that the current scale does not support claims of fully reliable, universal audio geo-localization. Instead, our results should be interpreted as evidence that modern audio-language models exhibit non-trivial, uneven, and fragile geo-inference capabilities under constrained conditions. We view this dataset as a first diagnostic benchmark rather than a definitive measure, and we explicitly position scaling, rebalancing, and broader environment coverage as key future work.

(4) On potential contamination from model pretraining data.
This is an important concern. We believe the risk of direct memorization is limited for several reasons. Aporee enforces strict usage policies that restrict automated scraping by AI systems, and, critically, even if audio files were accessed, obtaining reliable audio–location metadata pairings at scale is non-trivial. This lack of existing benchmarks is precisely why we constructed the dataset. Nevertheless, we do not claim to fully rule out partial exposure. Rather than assuming a clean separation, we frame our benchmark as measuring effective capability under realistic training conditions, which may include incidental exposure to similar content. Distinguishing memorization from genuine reasoning is an important open problem, and our error analyses (e.g., failures on acoustically similar but geographically distinct locations) suggest that models are far from exhibiting simple lookup-based behavior.