Unverified Commit 5f0fd118 authored by Jiahao Li's avatar Jiahao Li Committed by GitHub
Browse files

Optimize Qwen2VL vision model by precomputing cos/sin embeds before ViT blocks (#35837)

* Optimize Qwen2VL vision model by precomputing cos/sin embeds before ViT blocks

* Make rotary_pos_emb optional & fix type

* Adapt pre-computed cos/sin to Qwen2.5VL

* More concise
parent d72642bc
main 35597_custom_tokenizer action_to_notify_new_model_push_backup add-deci-lm add-dia add_eagle add_kernelize all_jobs_can_compare_against_prev_runs_clean_trigger base-model-loading batched_handle_empty_string build_ci_docker_image_amd2 change-mi250-ci-slack-channel change_build_input_tests change_to_draft_2 change_to_draft_3 change_to_draft_4 change_to_draft_4-release chat-template-quick-fix check-v4.49-release check_circleci_new_trigger check_circleci_tokenizer check_doc_image check_draft_4 check_env_runner check_push check_temp check_test_from_pretrained_low_cpu_mem_usage_equal check_torch_27 ci-test-huggingface-hub-0.29.0.rc6 ci-test-huggingface-hub-0.30.0.rc1 ci-test-huggingface-hub-v0.29.0.rc0 ci-test-huggingface-hub-v0.29.0.rc1 ci-test-huggingface-hub-v0.29.0.rc2 ci-test-huggingface-hub-v0.29.0.rc5 ci-test-huggingface-hub-v0.29.0.rc7 ci-test-huggingface-hub-v0.29.3.rc0 ci-test-huggingface-hub-v0.30.0.rc3-release ci-test-huggingface-hub-v0.31.0.rc0-release ci-test-huggingface-hub-v0.32.0.rc0-release ci-test-huggingface-hub-v0.32.0.rc1-release ci_with_commit_41b9b92b52215bed472c9a534a06abbc3a9a95cd ci_with_torch_2.7 ci_with_torch_2.7.1_commit_0ef339ff1b63bb03a388c79bfbebec9085e10564 ci_with_torch_2.7_commit_0ef339ff1b63bb03a388c79bfbebec9085e10564 ci_with_torch_version_base clean-modeling composable-tp continuous-batching custom-compute-loss-num-batches debug+_audio dep_create_token_type_id dependabot/pip/examples/flax/vision/torch-2.6.0 dependabot/pip/examples/tensorflow/language-modeling-tpu/transformers-4.50.0 disable-mi210-ci dummy-pr elie-temp-nope feature/#35425 find-test-failure-diff-between-envs fix-apex fix-compressed-tensors fix-doc-builder fix-flash-attention-with-static-cache fix-gemma3-grad-acc fix-modular fix-quantizer fix-tp-check fix/default_cb_scheduler fix_batch_test fix_circleci_not_triggered fix_module_conversion_util_ci fix_offload_disk_gguf fix_print fix_require_class fix_sam_samhq fix_tie3 fix_tiny_gh fixing_gptq_tests fsdp2-checkpointing get-our-efficiency-back glm4 gpt2 hf-papers image-chunked-prefill llama4-unhardcode merging_to_test metadata_job_2 mistral3-xpu-cpu-offload more-cleaning more_info_ci_temp muellerzr-fixup-warning multiple-modular new_blt no-more-pointing-at-remote-repos non-model-inits nouamane/context-parallel one-class-to-rule-them-all parallel pcuenca-patch-1 prefill-chunking push-ci-image raise-from random_dispatch refactor-from-pretrained-base-commit remove_unused_test_attribs revert-37178-revert-loadibng-issue run_amd_scheduled_ci_caller run_ci_without_kenlm skip_flaky_test skip_flaky_tests_double_check skip_internvl_tests slight-readme-reword spm_converter stop_repeating_setup temp123 test-datasets-main test-fused-moe test-tp-old-version test_fast_only_refactor tests-fetcher-test-all tok_refactor tokenizers_prerelease tp-test transformers-should-not-set-env-vars trigger_688f4707bfc5f6adc6f4f18c2081c5a66db590d1 trigger_all trigger_all_2 trigger_build trigger_doc_build_after_bot_push trigger_via_api_backup try_cpu_offload try_torch_2.7_on_circleci_jobs update-from-pretrained update-min-safetensors update-notification-service-amd-ci update-patch-helper update-recommended-reviewers update-tp-nits update_loss use_uv v4.49-release v4.49.0-AyaVision-release v4.49.0-Gemma-3-release v4.49.0-Mistral-3-release v4.49.0-SigLIP-2-release v4.49.0-SmolVLM-2-release v4.50-release v4.50.3-DeepSeek-3-release v4.51-release v4.51.3-BitNet-release v4.51.3-CSM-release v4.51.3-D-FINE-release v4.51.3-GraniteMoeHybrid-release v4.51.3-InternVL-release v4.51.3-Janus-release v4.51.3-LlamaGuard-release v4.51.3-MLCD-release v4.51.3-Qwen2.5-Omni-release v4.51.3-SAM-HQ-release v4.51.3-TimesFM-release v4.52-release vas-bert-attn-refactor vas-bert-attn-refactors vas-whisper-attn-refactor vb/add-baichuan vision_visualizer why_no_trigger working working-version v4.52.3 v4.52.2 v4.52.1 v4.52.0 v4.51.3 v4.51.3-TimesFM-preview v4.51.3-SAM-HQ-preview v4.51.3-Qwen2.5-Omni-preview v4.51.3-MLCD-preview v4.51.3-LlamaGuard-preview v4.51.3-Janus-preview v4.51.3-InternVL-preview v4.51.3-GraniteMoeHybrid-preview v4.51.3-D-FINE-preview v4.51.3-CSM-preview v4.51.3-BitNet-preview v4.51.2 v4.51.1 v4.51.0 v4.50.3 v4.50.3-DeepSeek-3 v4.50.2 v4.50.1 v4.50.0 v4.50.r32 v4.50.r3 v4.49.0 v4.49.0-SmolVLM-2 v4.49.0-SigLIP-2 v4.49.0-Mistral-3 v4.49.0-Gemma-3 v4.49.0-AyaVision
No related merge requests found
Showing with 207 additions and 72 deletions
+207 -72
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment