Handling Missing Pressure Levels (150 & 600 hPa) in AIFS_ens_v1 Training — ECMWF Archive Limitation
Hi guys,
First of all, thank you for releasing and maintaining the AIFS_ens_v1 model — it’s been really interesting to explore.
I have a question regarding the pressure level inputs used during training. The model documentation mentions that AIFS_ens_v1 uses 13 pressure levels, but the ECMWF archived data (up to 2023-04-27) only includes 11 pressure levels — specifically, the 150 hPa and 600 hPa levels are missing.
Could you please clarify how this was handled during training?
Understanding this would help me preprocess ECMWF data consistently for reproducing or fine-tuning AIFS_ens_v1 forecasts.
Thanks again for your great work and for taking the time to clarify! 🙏
Hi,
I'm not sure what archive you are referencing, but as you can see from the opendata request, those pressure levels are produced.
Therefore, we at ECMWF have access to that data for training.
That data is even accessible on the CDS
Hi, thank you for your response.
I am talking about the ecmwf archive-catalogue. Here's the link for your reference:
https://apps.ecmwf.int/archive-catalogue/?stream=enfo&levtype=pl&time=00%3A00%3A00&expver=1&month=jan&year=2022&date=2022-01-01&type=pf&class=od
You can see here only 11/13 pressure levels are present.
For training you needn't need the pf, nor do we provide our full operational archive openly
https://apps.ecmwf.int/archive-catalogue/?class=od&stream=enfo&expver=1&type=cf&year=2022&month=jan&levtype=pl&date=2022-01-01&time=00:00:00
@wangsherpa
thanks for your discussion. To add to Cookie's comment. The current AIFS ENS doesn't train on ensemble data, but only uses ensemble data in inference.
This is discussed briefly in the paper
https://arxiv.org/abs/2412.15832
Thanks a lot for the clarification — that helps!
We’re currently performing inference on the archived forecast data, but we noticed that the 150 hPa(present in cf) and 600 hPa levels are missing in the ECMWF archive.
Does it make sense to interpolate these missing pressure levels, and if so, what would be the recommended way to do it?
Hi, would it make sense to use Metview for interpolation? I'm also applying AIFS ENS on historical data from the archive.