Repository navigation
Add DeBERTa-v2 and DeBERTa-v3 base encoder support - #466
Conversation
| "position_to_content" | ||
| ] do | ||
| @variant variant | ||
| test "base encoder matches Python for #{@variant}" do |
There was a problem hiding this comment.
Please replicate the test structure like we do in other models, with one test per architecture. Also, instead of generating model files into the repo, let's use checkpoints from HuggingFace. Here are small test checkpoints that you can use. The references values should come from Python run, see this README section for more details.
There was a problem hiding this comment.
Done. Now we are using the Hugging Face checkpoint and the Python references
| @@ -0,0 +1,123 @@ | |||
| defmodule Bumblebee.Text.DebertaV2.DisentangledAttention do | |||
There was a problem hiding this comment.
There is no need for a separate module, we can keep that within the model module. However, ideally we would use the high-level Transformer.blocks/2. Perhaps there is a way we can support this attention structure with an option, worth letting the agent analyze and try.
There was a problem hiding this comment.
Changed to use Transformer.blocks/2 with the remaining helpers inside the model
| "encoder.conv_norm" => "deberta.encoder.conv.LayerNorm" | ||
| } | ||
|
|
||
| Enum.reduce(@for.projection_names(spec), mapping, fn projection, mapping -> |
There was a problem hiding this comment.
projection_names can be a private function here. Is there a reason we need it to be this dynamic though? Typically we just list all the keys that our implementation uses into their counterparts. Each architecture only uses a subset of the keys but that's not a problem.
There was a problem hiding this comment.
I replaced the helper with explicit mappings
|
Thanks. I will do the changes. |
| cross_attention = fn hidden_state -> | ||
| {hidden_state, cross_attention, cross_attention_cache, _cross_attention_relative_bias} = | ||
| multi_head_attention(hidden_state, cross_hidden_state, cross_hidden_state, | ||
| attention_fun.(hidden_state, cross_hidden_state, cross_hidden_state, |
There was a problem hiding this comment.
This PR is a good indicator that we should revisit this and have the model compose from smaller blocks rather than have everything driven by options. I will do it in a separate PR though, this is good :)
|
Thank you! |
Add the DeBERTa-v2/v3 base encoder as
Bumblebee.Text.DebertaV2, with Hugging Face configuration/parameter mappings and DeBERTa Unigram tokenizer support.The encoder supports absolute and relative positions, shared or separate position projections, content-to-position and position-to-content attention, logarithmic position buckets, optional convolution, padding masks, hidden states, and attention outputs. Tiny local checkpoints and their Python generation script cover five configurations.
Validation: all seven encoder, tokenizer, and configuration tests passed on macOS ARM64 and Linux x86_64. Outputs are compared with Python references, and loading checks reject missing, unused, or incorrectly shaped parameters. The integration runs used elixir-nx/nx#1842 and elixir-nx/axon#681. The encoder was also exercised with the real GLiNER 2.5 base checkpoint in the dependent entity-extraction contribution.
AI disclosure: most of this implementation was generated with an AI coding agent.