Create README.md

README.md

@@ -0,0 +1,82 @@
+---
+license: mit
+datasets:
+- openai/summarize_from_feedback
+base_model:
+- google/gemma-7b
+---
+# Model Card for MA-RLHF
+<a href="https://iclr.cc/Conferences/2024" target="_blank">
+      <img alt="ICLR 2025" src="https://img.shields.io/badge/Proceedings-ICLR2025-red" />
+</a>
+<a href="https://github.com/ernie-research/MA-RLHF" target="_blank">
+      <img alt="Github" src="https://img.shields.io/badge/Github-MA_RLHF-green" />
+   </a>
+
+This repository contains the official checkpoint for [Reinforcement Learning From Human Feedback with Macro Actions (MA-RLHF)](https://arxiv.org/pdf/2410.02743). 
+
+## Model Description
+
+MA-RLHF is a novel framework that integrates macro actions into conventional RLHF. The macro actions are sequences of tokens or higher-level language constructs, with can be computed through different defined termination conditions, like n-gram based, perplexity-based, or parsing-based termination conditions. By introducing macro actions into RLHF, we reduce the number of decision points and shorten decision trajectories, alleviating the credit assignment problem caused by long temporal distances.
+
+
+|Model|Checkpoint|Base Model|Dataset| 
+|-----|----------|-|-|
+|TLDR-Gemma-2B-MA-PPO-Fixed5|🤗 [HF Link](https://huggingface.co/baidu/TLDR-Gemma-2B-MA-PPO-Fixed5)|[google/gemma-2b](https://huggingface.co/google/gemma-2b)|[openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)
+|TLDR-Gemma-7B-MA-PPO-Fixed5|🤗 [HF Link](https://huggingface.co/baidu/TLDR-Gemma-7B-MA-PPO-Fixed5)|[google/gemma-7b](https://huggingface.co/google/gemma-7b)|[openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)
+|TLDR-Gemma-2-27B-MA-PPO-Fixed5|🤗 [HF Link](https://huggingface.co/baidu/TLDR-Gemma-2-27B-MA-PPO-Fixed5)|[google/gemma-2-27b](https://huggingface.co/google/gemma-2-27b)|[openai/summarize_from_feedback](https://huggingface.co/datasets/openai/summarize_from_feedback)
+|HH-RLHF-Gemma-2B-MA-PPO-Fixed5|🤗 [HF Link](https://huggingface.co/baidu/HH-RLHF-Gemma-2B-MA-PPO-Fixed5) |[google/gemma-2b](https://huggingface.co/google/gemma-2b)|[Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf)
+|HH-RLHF-Gemma-7B-MA-PPO-Fixed5|🤗 [HF Link](https://huggingface.co/baidu/HH-RLHF-Gemma-7B-MA-PPO-Fixed5) |[google/gemma-7b](https://huggingface.co/google/gemma-7b)|[Dahoas/full-hh-rlhf](https://huggingface.co/datasets/Dahoas/full-hh-rlhf)
+|APPS-Gemma-2B-MA-PPO-Fixed10|🤗 [HF Link](https://huggingface.co/baidu/APPS-Gemma-2B-MA-PPO-Fixed10) |[google/codegemma-2b](https://huggingface.co/google/codegemma-2b)|[codeparrot/apps](https://huggingface.co/datasets/codeparrot/apps)
+|APPS-Gemma-7B-MA-PPO-Fixed10|🤗 [HF Link](https://huggingface.co/baidu/APPS-Gemma-7B-MA-PPO-Fixed10) |[google/codegemma-7b-it](https://huggingface.co/google/codegemma-7b-it)|[codeparrot/apps](https://huggingface.co/datasets/codeparrot/apps)
+
+
+## Model Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_path = "baidu/TLDR-Gemma-7B-MA-PPO-Fixed5"
+
+tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+
+model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", torch_dtype='auto', trust_remote_code=True)
+
+input_text = """
+POST Subreddit: r/cats
+Hello everyone! One of my cats is about 10 years old now, she is pretty much strictly
+indoors save for some time she spends on our screened in porch each day. (She likes
+to watch the birds in the yard while she suns herself by the pool, quite the princess).
+Anyway, when she was younger she was very active and quite small, however with
+age she has put on a pretty hefty amount of weight. I feed her indoor cat food
+for weight control, I’ve switched brands a few times trying to find something that
+works, I’ve cut back on feeding her by a lot (she gets very angry and demanding
+when she wants food but I don’t give in) however, nothing really seems to work.
+I’ve tried cat toys, and bought a harness thinking I could try to walk her but she just
+lays down and looks at me like I’m stupid. Basically I just want to know if you all
+have any suggestions for exercise or food. I care about her and don’t want this to
+get any worse. I also have another cat that eats the same amount and type of food
+as her and is a completely normal weight and only a year younger, however he is a
+male, not sure if that makes a difference in predisposition for weight gain. They are
+also both fixed. TL;DR: 
+"""
+
+input_ids = tokenizer(input_text, return_tensors='pt').to(model.device)
+output_ids = model.generate(**input_ids, max_new_tokens=20)
+response = tokenizer.decode(output_ids[0], skip_special_tokens=True)
+
+print(response)
+```
+
+## Citation
+
+```
+@inproceedings{
+  chai2025marlhf,
+  title={{MA}-{RLHF}: Reinforcement Learning from Human Feedback with Macro Actions},
+  author={Yekun Chai and Haoran Sun and Huang Fang and Shuohuan Wang and Yu Sun and Hua Wu},
+  booktitle={The Thirteenth International Conference on Learning Representations},
+  year={2025},
+  url={https://openreview.net/forum?id=WWXjMYZxfH}
+}
+```