Skip to content

De ITN: decouple cardinal grammar from TN - #487

Open
adelina23dunina wants to merge 10 commits into
NVIDIA:staging/de_tn_itn_v2from
adelina23dunina:de_itn_cardinal_decoupled_v2
Open

adelina23dunina wants to merge 10 commits into
NVIDIA:staging/de_tn_itn_v2from
adelina23dunina:de_itn_cardinal_decoupled_v2

Conversation

@adelina23dunina

Copy link
Copy Markdown

What does this PR do ?

Rebuilds the German ITN cardinal grammar as a standalone WFST with its own TSV data, so it no longer depends on inverting the German TN cardinal tagger and verbalizer.

Details:

  • CardinalFst in both the tagger and the verbalizer now take no arguments. The grammar is
    built from new tables in de/data/cardinal/ (zero, digits, irregular_teens, teens,
    tens) instead of from tn_cardinal_tagger / tn_cardinal_verbalizer.
  • Adds de/graph_utils.py and de/utils.py to the ITN submodule.
  • Output now carries German thousands separators, e.g. eintausend -> 1.000 rather than 1000.
  • Numbers below thirteen are still left spelled out, as before.
  • Keeps the attributes the other German semiotic classes read off the cardinal tagger, so
    date/decimal/fraction/measure/money/ordinal/telephone are unaffected:
    graph_no_exception, graph_hundred_component_at_least_one_none_zero_digit,
    optional_minus_graph.
  • Bumps DE_TN_CACHE in the Jenkinsfile so CI rebuilds the grammars instead of reusing
    stale compiled FARs.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here. — updated tests/nemo_text_processing/de/data_inverse_text_normalization/test_cases_cardinal.txt, which feeds both test_cardinal.py and test_sparrowhawk_inverse_text_normalization.sh
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files? — added de/data/__init__.py and de/data/cardinal/__init__.py
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done. — none present in inverse_text_normalization/de/
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo). — n/a, no new language and no user-facing API change
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py. — n/a, German is already registered there

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Comment thread nemo_text_processing/inverse_text_normalization/de/taggers/cardinal.py Outdated
Comment thread nemo_text_processing/inverse_text_normalization/de/taggers/cardinal.py Outdated
Comment thread nemo_text_processing/inverse_text_normalization/de/verbalizers/cardinal.py Outdated
Comment thread nemo_text_processing/inverse_text_normalization/de/graph_utils.py Outdated
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
from nemo_text_processing.inverse_text_normalization.de.utils import get_abs_path, load_labels


AND = "und"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tsv file instead of hardcoding

tens = pynini.string_file(get_abs_path("data/cardinal/tens.tsv"))
ties = tens + pynutil.insert("0")
# German flips ones and tens in two-digit numbers. The WFST below handles these flips.
delete_und = pynutil.delete(AND)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's use the closure instead of adding .ques

@@ -0,0 +1,42 @@
# Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

year

Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
@adelina23dunina
adelina23dunina force-pushed the de_itn_cardinal_decoupled_v2 branch from 61c1ad8 to 6fb8682 Compare September 15, 2026 17:46
Signed-off-by: Adelina Dunina <adunina@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants