Hi TN Roman: Expanding the numeral support for roman class till 3999 with compositional approach

[shrpawar-alt]

What does this PR do ?

Expanding the numeral support for roman class till 3999 with compositional approach

Before your PR is "Ready for review"

Pre checks:

• Have you signed your commits? Use git commit -s to sign.
• Do all unittests finish successfully before sending PR?
  1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
  2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
• If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
• Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
• Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
• Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
• If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
• Remove import guards (try import: ... except: ...) if not already done.
• If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
• Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

• New Feature
• Bugfix
• Documentation
• Test

If you haven't finished some of the above items you can still open "Draft" PR.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

[folivoramanh]

resolve the merge conflict

[folivoramanh]

how to make sure that this is really roman? for example if only 1 character appeared (V, I, L, D, C, X, M) how to know that it's really roman numerals but not English alphabet (might be codeswitch...?)

[shrpawar-alt]

While this is a valid concern, it represents a necessary trade-off. Currently, the grammar relies on adjacency: single characters (I, V, X) are only evaluated as Roman numerals if they are directly adjacent to a Hindi word (with or without a hyphen). If they appear in isolation, they safely fall back to the English alphabet. While this does leave a rare edge case for code-switched English letters (e.g., 'विटामिन C'), trying to fix it will introduce worse regressions. If we restrict length to 2+ characters, we will break valid cases like 'विश्व युद्ध-I' or 'खंड V'. If we use a whitelist of trigger words, we would lose generalization for thousands of valid proper nouns (like satellites or mission names). Considering that Roman numerals overwhelmingly appear adjacent to Hindi words, I think accepting the rare code-switched alphabet character as a limitation is the safest approach here.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.