Add hf_infer script for inference using HuggingFace backend

[WinterShiver]

What does this PR do?

Add hf_infer script for inference using HuggingFace backend. This script is developed referring to vllm_infer.

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

[gemini-code-assist]

Summary of Changes

Hello @WinterShiver, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new Python script, hf_infer.py, which enables batch inference using models powered by the Hugging Face Transformers backend. The script provides a flexible and efficient way to generate predictions for a given dataset, supporting both base models and those fine-tuned with PEFT adapters. It integrates seamlessly with the llamafactory's data loading and templating system, offering a comprehensive solution for evaluating models or generating outputs on a large scale.

Highlights

• New Inference Script: Introduces a new Python script, hf_infer.py, specifically designed for performing batch inference using models from the Hugging Face Transformers library.
• Hugging Face Backend Integration: The script leverages the Hugging Face ecosystem for model loading (AutoModelForCausalLM), tokenization (AutoTokenizer), and generation (GenerationConfig), ensuring compatibility with a wide range of pre-trained models.
• PEFT Adapter Support: Allows for the loading of Parameter-Efficient Fine-Tuning (PEFT) adapters alongside the base model, enabling inference with fine-tuned models without merging the adapter weights.
• Batch Generation and Output: Supports batch processing of datasets for efficient generation and saves the generated prompts, predictions, and labels to a JSONL file for easy analysis.
• Configurable Generation Parameters: Provides extensive control over the generation process through various parameters such as temperature, top_p, top_k, max_new_tokens, and repetition_penalty.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new script hf_infer.py for running inference using the HuggingFace backend. The script is well-structured and follows the pattern of similar scripts in the repository.

My review focuses on improving security, configurability, and maintainability. I've pointed out a security risk with trust_remote_code being hardcoded, suggested making torch_dtype and generation parameters more configurable, and noted an unused seed parameter that should be used to ensure reproducibility.

Overall, this is a great addition. Addressing these points will make the script more robust and user-friendly.

[gemini-code-assist]

Hardcoding torch_dtype to torch.bfloat16 can lead to compatibility issues on hardware that doesn't support it. It's better to make this configurable. Consider adding an infer_dtype argument to the function (e.g., with a default of 'auto') and then dynamically selecting the appropriate torch.dtype based on hardware support (torch.cuda.is_bf16_supported()) or the user's explicit choice.

[gemini-code-assist]

The GenerationConfig is created manually, which is redundant as get_infer_args already returns a generating_args object with these parameters. This generating_args object is currently unused. To simplify the code and avoid redundancy, you should use the generating_args object directly. This also allows for more flexible generation settings, such as do_sample, to be configured via arguments.

    gen_cfg = GenerationConfig(**generating_args.to_dict())