Quentin/DoLuEdu-WordFileParse
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

init: WordFileParse docx→HTML 解析服务（含 Word EQ 域公式修复）

Browse Source 
服务：Spring Boot 3.2 + docx4j 8.3.9，POST /api/v3/server_office_file/file/parse
      接受 .docx，返回 HTML（公式 → LaTeX，图片 → RustFS 外链）

公式支持三类来源：
- OMML（Word 2007+ 公式）      → Python omml_mathml2latex
- MathType OLE 对象            → Ruby mathtype2mathml → MathML → LaTeX
- Word EQ 域（老教辅主力）      → EqToLatexUtil（本次新增）

本次改动：
1. 新增 java-svc/.../core/EqToLatexUtil.java：
   移植自 jby backend-services office-core，EQ 语法 → LaTeX
   （\f→\dfrac、\r→\sqrt、\i→\int/\sum/\prod、\s→^{}/_{}、\b→\left\right、
     \a→array、\x→\boxed/\overline、\o\al 堆叠、圈码、化学符号、极限、连等式）
   品牌标记 JBYU* → WPU*，与采集端一致
2. CtoMathFinder：接上 EQ 分支（原为 not-ported 桩），域代码经
   EqToLatexUtil.convert 转 LaTeX 写入 eqLatexMap（WPEQNUM 占位符）
3. WordParseUtil.dealPlaceholder：EQ 的 LaTeX 同样加 \( \) 包裹
   （原先因 WPEQNUM 例外不加包裹，前端/MathJax 不渲染）

修前症状：教辅 Word 里分式/方程组/上下标整段丢失（如 {−1/3, 1} 变成 {－，1}）
修后实测：46 份教辅 2908 处 EQ 域全部还原，公式总数 2954 处、零未包裹残留

安全：application.yml（含 RustFS 密钥）不入库，提供 application.yml.example；
      OssService 的 @Value 默认密钥已清空
This commit is contained in:
Quentin
2026-09-29 09:43:13 +08:00
commit 3271fee5cc
360 changed files with 86082 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

68
scripts/omml_mathml2latex/utils/__init__.py Normal file
View File

@@ -0,0 +1,68 @@
"""
Utility functions for the refactor package.
"""
from .color_helpers import (
detect_uniform_color,
hex_to_latex_color,
should_suppress_color,
wrap_with_uniform_color,
wrap_element_with_color,
)
from .xml_utils import (
text_content,
local_name,
decode_html_entities,
extract_math_equations,
)
from .helpers import (
is_default_color,
)
from .text_processing import (
write_text,
read_text,
convert_spaces_to_latex,
escape_latex_special_chars,
)
from .timer import (
PerformanceTimer,
TimingRecord,
TimingSummary,
get_global_timer,
enable_global_timer,
disable_global_timer,
)
from .debug import (
is_debug_enabled,
debug_print,
)
__all__ = [
'detect_uniform_color',
'hex_to_latex_color',
'should_suppress_color',
'wrap_with_uniform_color',
'wrap_element_with_color',
'text_content',
'local_name',
'decode_html_entities',
'extract_math_equations',
'is_default_color',
'write_text',
'read_text',
'convert_spaces_to_latex',
'escape_latex_special_chars',
'PerformanceTimer',
'TimingRecord',
'TimingSummary',
'get_global_timer',
'enable_global_timer',
'disable_global_timer',
'is_debug_enabled',
'debug_print',
]

296
scripts/omml_mathml2latex/utils/chemistry.py Normal file
View File

@@ -0,0 +1,296 @@
"""
Chemistry detection utilities for identifying chemical equations and contexts.
This module provides functions to detect chemical reactions, conditions, and
other chemistry-specific patterns in mathematical equations.
"""
from typing import Optional, Dict
from lxml import etree as ET
# OMML namespace constant
OMML_NS = "http://schemas.openxmlformats.org/officeDocument/2006/math"
def contains_chinese_characters(text: str) -> bool:
"""Check if text contains Chinese characters.
Args:
text: Text to check
Returns:
True if text contains at least one Chinese character
Example:
>>> contains_chinese_characters('加热')
True
>>> contains_chinese_characters('heat')
False
"""
if not text:
return False
for char in text:
if '\u4e00' <= char <= '\u9fff': # Chinese character range
return True
return False
def is_typical_reaction_condition_text(text: str) -> bool:
"""Check if text looks like a typical chemical reaction condition.
Args:
text: Text to check
Returns:
True if text contains typical chemical reaction condition terms
Example:
>>> is_typical_reaction_condition_text('加热')
True
>>> is_typical_reaction_condition_text('heat')
True
>>> is_typical_reaction_condition_text('abc')
False
"""
if not text:
return False
# Common chemical reaction conditions
typical_conditions = {
# Chinese conditions
"电解", "加热", "催化剂", "高温", "高压", "光照", "加压", "蒸馏", "熔融", "溶解", "酸化", "碱化",
"氧化", "还原", "分解", "合成", "中和", "干燥", "过滤", "沉淀", "结晶", "升华", "煅烧",
# English conditions
"heat", "catalyst", "pressure", "light", "oxidation", "reduction", "electrolysis",
"distillation", "crystallization", "precipitation", "filtration", "drying",
# Symbols and abbreviations
"Δ", "hv", "H+", "OH-", "cat", "pt", "ni", "pd"
}
text_lower = text.lower().strip()
return any(condition.lower() in text_lower for condition in typical_conditions)
def is_chemical_reaction_arrow(eqArr_node: ET._Element) -> Optional[Dict[str, str]]:
"""Detect if an eqArr represents a chemical reaction arrow with conditions.
Args:
eqArr_node: OMML eqArr element
Returns:
Dictionary with chemical reaction info if detected, None otherwise.
Structure: {
'above': str, # Text above the arrow (e.g., '电解')
'below': str, # Text below the arrow (e.g., '加热', can be empty)
'type': str # 'chemical_reaction'
}
Example:
>>> # Assuming eqArr_node is an OMML element with reaction conditions
>>> result = is_chemical_reaction_arrow(eqArr_node)
>>> result['above']
'电解'
"""
# Find all equation array elements
e_elements = eqArr_node.findall(f"{{{OMML_NS}}}e")
# Must have exactly 2 elements for chemical reaction conditions
if len(e_elements) != 2:
return None
# Extract text from first element (above the arrow)
above_text = ""
first_elem = e_elements[0]
# Extract text from all t elements in the first element, avoiding duplication
# Use a set to track unique text content
text_parts = set()
for t_elem in first_elem.findall(f".//{{{OMML_NS}}}t"):
if t_elem.text and t_elem.text.strip():
text_parts.add(t_elem.text.strip())
# Join unique text parts
above_text = "".join(sorted(text_parts))
# Extract text from second element (below the arrow)
below_text = ""
second_elem = e_elements[1]
# Extract text from all t elements in the second element, avoiding duplication
text_parts_below = set()
for t_elem in second_elem.findall(f".//{{{OMML_NS}}}t"):
if t_elem.text and t_elem.text.strip():
text_parts_below.add(t_elem.text.strip())
# Join unique text parts
below_text = "".join(sorted(text_parts_below))
# Validate that this looks like a chemical reaction pattern
if above_text.strip():
# Check if contains chemical condition patterns (Chinese text or typical reaction words)
if (contains_chinese_characters(above_text) or
is_typical_reaction_condition_text(above_text) or
contains_chinese_characters(below_text) or
is_typical_reaction_condition_text(below_text)):
return {
'above': above_text.strip(),
'below': below_text.strip(),
'type': 'chemical_reaction'
}
return None
def detect_chemistry_context(equation_text: str) -> bool:
"""Detect if equation suggests chemistry context - Conservative approach.
Args:
equation_text: The full equation text to analyze
Returns:
True if chemistry context is detected
Example:
>>> detect_chemistry_context('H2O + pH = 7')
True
>>> detect_chemistry_context('sin x + cos x')
False
"""
if not equation_text:
return False
# Check if we're in a set theory context - if so, don't treat letters as chemical elements
if has_set_theory_symbols(equation_text):
return False
# Check if we're in a mathematical context (trigonometry, calculus, etc.)
if has_mathematical_functions(equation_text):
return False
# Conservative chemistry detection: require multiple indicators, not just chemical elements
chemistry_indicators = ['pH', 'pOH', 'mol', 'atm', 'bar', 'Torr', 'eq', 'aq', 'sol',
'↑', '↓', '→', '⇋', '⇌', 'xlongequal', 'rightarrow']
# Only return True if we have actual chemistry indicators in the equation text
return any(indicator in equation_text for indicator in chemistry_indicators)
def has_mathematical_functions(equation_text: str) -> bool:
"""Check if the equation contains mathematical functions.
Args:
equation_text: The equation text to analyze
Returns:
True if mathematical functions are detected
Example:
>>> has_mathematical_functions('sin x + cos x')
True
>>> has_mathematical_functions('H2O')
False
"""
if not equation_text:
return False
# Common mathematical functions that indicate math context, not chemistry
math_functions = ['sin', 'cos', 'tan', 'log', 'ln', 'exp', 'sqrt', 'lim', 'max', 'min',
'arcsin', 'arccos', 'arctan', 'sinh', 'cosh', 'tanh']
return any(func in equation_text for func in math_functions)
def has_set_theory_symbols(equation_text: str) -> bool:
"""Check if the equation contains set theory symbols.
Args:
equation_text: The equation text to analyze
Returns:
True if set theory symbols are detected
Example:
>>> has_set_theory_symbols('A ∩ B ⊂ C')
True
>>> has_set_theory_symbols('H2O')
False
"""
if not equation_text:
return False
set_symbols = ['⊂', '⊃', '⊆', '⊇', '⊄', '⊅', '⊈', '⊉', '⊊', '⊋',
'∈', '∉', '∋', '∌', '∪', '∩', '∖', '∁']
return any(symbol in equation_text for symbol in set_symbols)
def detect_physics_context(equation_text: str) -> bool:
"""Detect if equation suggests physics context.
Args:
equation_text: The equation text to analyze
Returns:
True if physics context is detected
Example:
>>> detect_physics_context('F = ma, where m is in kg')
True
>>> detect_physics_context('x + y = z')
False
"""
if not equation_text:
return False
# Check if we're in a mathematical context (trigonometry, calculus, etc.)
if has_mathematical_functions(equation_text):
return False
# Common physics constants and units - look for these in the equation context
physics_indicators = [
'kg', 'm', 's', 'A', 'K', 'mol', 'cd', # SI base units
'Hz', 'N', 'Pa', 'J', 'W', 'C', 'V', 'F', 'Ω', 'S', 'Wb', 'T', 'H', # Derived units
'eV', 'cal', 'Bar', 'L', # Common non-SI
# Note: removed single-letter constants like 'c', 'h', 'k', 'G', 'R', 'NA'
# These are too common in math contexts and cause false positives
]
# Only return True if we have actual physics indicators in the equation text
return any(indicator in equation_text for indicator in physics_indicators)
def is_physics_constant(content: str) -> bool:
"""Check if content is a known physics constant.
Args:
content: String to check
Returns:
True if content represents a known physics constant
Example:
>>> is_physics_constant('c')
True
>>> is_physics_constant('x')
False
"""
constants = {
'c', # speed of light
'h', # Planck constant
'hbar', # reduced Planck constant
'k', # Boltzmann constant
'kB', # Boltzmann constant
'G', # gravitational constant
'R', # gas constant
'NA', # Avogadro number
'e', # elementary charge
'me', # electron mass
'mp', # proton mass
'mn', # neutron mass
'mu', # atomic mass unit
'epsilon0', # vacuum permittivity
'mu0', # vacuum permeability
'alpha', # fine structure constant
'sigma', # Stefan-Boltzmann constant
}
return content in constants

259
scripts/omml_mathml2latex/utils/chemistry_detection.py Normal file
View File

@@ -0,0 +1,259 @@
"""
Chemistry equation detection utilities.
This module provides functions to detect whether an equation is chemistry-related
based on strong indicators like multi-char elements, charges, and reaction arrows.
"""
from lxml import etree as ET
from typing import Optional, Tuple
from ..data.chemistry_data import (
MULTI_CHAR_ELEMENTS,
ALL_CHEMICAL_ELEMENTS,
CHEMICAL_GROUPS,
CHINESE_CHEMICAL_TERMS,
)
def detect_chemistry_equation(root: ET.Element) -> bool:
"""
Detect if equation is chemistry-related using STRONG indicators only.
DISABLED: Subject detection is obsolete. We rely on mathvariant="normal" only.
Chemistry formulas are properly formatted when MathML includes mathvariant="normal".
Args:
root: The root MathML element (<math>)
Returns:
False - always (chemistry detection disabled)
"""
# DISABLED: Always return False - no heuristic chemistry detection
return False
# OLD CODE (kept for reference):
# Strong indicators:
# 1. Multi-char chemical elements in single <mi> tag (Cu, Al, Fe, Mn, etc.)
# 2. Chemical groups (OH, SO, NO, CO, NH)
# 3. Element + numeric subscript (H₂O pattern)
# 4. Electric charges in superscript (^{2+}, ^{2-})
# 5. Chemical arrows (↑, ↓)
# 6. Chinese chemical terms (高温, 电解, etc.)
#
# for node in root.iter():
# # Remove namespace if present
# tag = node.tag.split('}')[-1] if '}' in node.tag else node.tag
# text = (node.text or '').strip()
#
# # Strong indicator 1: Multi-char element in single <mi>
# if tag == 'mi' and text in MULTI_CHAR_ELEMENTS:
# return True
#
# # Strong indicator 2: Chemical group
# if tag == 'mtext' and text in CHEMICAL_GROUPS:
# return True
#
# # Strong indicator 3: Element + numeric subscript (H₂O, SO₄ pattern)
# if tag == 'msub' and len(node) >= 2:
# base_node = node[0]
# sub_node = node[1]
# base_text = (base_node.text or '').strip()
# sub_text = (sub_node.text or '').strip()
#
# # Element with numeric subscript
# if base_text in ALL_CHEMICAL_ELEMENTS and sub_text.isdigit():
# return True
#
# # Strong indicator 4: Electric charge superscript (Cu^{2+}, Mn^{2-})
# if tag == 'msup' and len(node) >= 2:
# sup_node = node[1]
# sup_text = (sup_node.text or '').strip()
#
# # Check for charge notation (contains + or -)
# # Pattern: "2+", "+", "2-", "-", "3+", etc.
# if sup_text and ('+' in sup_text or '-' in sup_text):
# # Additional check: ensure it's likely a charge, not math
# # Charges are typically: digit* followed by +/-
# # Math would be expressions like "x+1", "y-2"
# # Simple heuristic: if it's short and has +/- at end, it's likely a charge
# if len(sup_text) <= 3 and (sup_text.endswith('+') or sup_text.endswith('-')):
# return True
# # Also check for format: "2 +" or "2 -" (with space)
# parts = sup_text.split()
# if len(parts) == 2 and parts[0].isdigit() and parts[1] in ('+', '-'):
# return True
#
# # Strong indicator 5: Chemical arrows (precipitate, gas)
# if tag == 'mo' and text in ('↑', '↓'):
# return True
#
# # Strong indicator 6: Chinese chemical terms
# if tag in ('mtext', 'mi') and text in CHINESE_CHEMICAL_TERMS:
# return True
#
# return False
def calculate_chemistry_probability(root: ET.Element, subject_mode: Optional[str] = None) -> float:
"""
Calculate chemistry probability score based on multiple indicators.
Scoring weights:
- Ion charge present (^{2+}, ^{3-}): +1.0
- Multi-char element in single <mi>: +0.8
- Nearby multi-char element: +0.8
- Reaction arrow (→, ⇌, ⇄): +0.6
- Chemical subscripts (H_2, O_2): +0.4
- Chinese chemical term: +0.3
- User set --subject chemistry: +0.5
Thresholds:
- Chemistry mode: ≥0.8 (merge fragmented elements)
- Auto-mode: ≥1.5 (only VERY strong indicators)
Args:
root: The root MathML element (<math>)
subject_mode: Optional subject mode ('chemistry', 'physics', etc.)
Returns:
Probability score (0.0-3.0+)
Examples:
>>> # M n^{2+} with MnO nearby: 1.0 + 0.8 + 0.8 = 2.6 (strong)
>>> calculate_chemistry_probability(math_root) # 2.6
>>> # Just "M n" with no context: 0.0 (weak)
>>> calculate_chemistry_probability(math_root) # 0.0
>>> # Chemistry mode boost: M n → 0.5 (mode) + context
>>> calculate_chemistry_probability(math_root, 'chemistry') # 0.5+
"""
# DISABLED: Always return 0.0 - no chemistry detection
return 0.0
# OLD CODE (kept for reference):
# score = 0.0
# # Mode boost: user explicitly set --subject chemistry
# if subject_mode == 'chemistry':
# score += 0.5
#
# # Counters for various indicators
# has_ion_charge = False
# multi_char_element_count = 0
# has_reaction_arrow = False
# chemical_subscript_count = 0
# has_chinese_term = False
#
# for node in root.iter():
# # Remove namespace if present
# tag = node.tag.split('}')[-1] if '}' in node.tag else node.tag
# text = (node.text or '').strip()
#
# # Indicator 1: Multi-char element in single <mi> (+0.8 each, max 2)
# if tag == 'mi' and text in MULTI_CHAR_ELEMENTS:
# multi_char_element_count += 1
#
# # Indicator 2: Chemical group (+0.8)
# if tag == 'mtext' and text in CHEMICAL_GROUPS:
# multi_char_element_count += 1
#
# # Indicator 3: Element + numeric subscript (H₂O pattern) (+0.4 each, max 3)
# if tag == 'msub' and len(node) >= 2:
# base_node = node[0]
# sub_node = node[1]
# base_text = (base_node.text or '').strip()
# sub_text = (sub_node.text or '').strip()
#
# if base_text in ALL_CHEMICAL_ELEMENTS and sub_text.isdigit():
# chemical_subscript_count += 1
#
# # Indicator 4: Ion charge superscript (+1.0, strongest indicator)
# if tag == 'msup' and len(node) >= 2:
# sup_node = node[1]
# sup_text = (sup_node.text or '').strip()
#
# # Check for charge notation
# if sup_text and ('+' in sup_text or '-' in sup_text):
# if len(sup_text) <= 3 and (sup_text.endswith('+') or sup_text.endswith('-')):
# has_ion_charge = True
# # Also check for format: "2 +" or "2 -"
# parts = sup_text.split()
# if len(parts) == 2 and parts[0].isdigit() and parts[1] in ('+', '-'):
# has_ion_charge = True
#
# # Indicator 5: Reaction arrows (+0.6)
# if tag == 'mo':
# # Chemical arrows (precipitate, gas)
# if text in ('↑', '↓'):
# has_reaction_arrow = True
# # Reaction arrows (→, ⇌, ⇄, etc.)
# if text in ('→', '←', '⇌', '⇄', '⟶', '⟵', '⇀', '↽'):
# has_reaction_arrow = True
#
# # Indicator 6: Chinese chemical terms (+0.3)
# if tag in ('mtext', 'mi') and text in CHINESE_CHEMICAL_TERMS:
# has_chinese_term = True
#
# # Calculate total score
# if has_ion_charge:
# score += 1.0
#
# # Cap multi-char elements at 2 (prevent over-scoring)
# score += min(multi_char_element_count, 2) * 0.8
#
# if has_reaction_arrow:
# score += 0.6
#
# # Cap chemical subscripts at 3
# score += min(chemical_subscript_count, 3) * 0.4
#
# if has_chinese_term:
# score += 0.3
#
# return score
def detect_chemistry_equation_with_probability(root: ET.Element) -> Tuple[bool, float]:
"""
Detect chemistry equation and return both boolean and probability score.
DISABLED: Subject detection is obsolete.
Args:
root: The root MathML element (<math>)
Returns:
Tuple of (False, 0.0) - always (chemistry detection disabled)
"""
# DISABLED: Always return (False, 0.0)
return False, 0.0
# OLD CODE (kept for reference):
# probability = calculate_chemistry_probability(root)
# is_chemistry = probability >= 1.0 # Strong indicator threshold
# return is_chemistry, probability
def is_chemistry_variable(text: str, in_chemistry_context: bool) -> bool:
"""
Determine if text is a chemistry variable (should stay italic).
DISABLED: Chemistry detection is obsolete.
Args:
text: The identifier text
in_chemistry_context: Whether we're in a chemistry equation (ignored)
Returns:
False - always (not used anymore)
"""
# DISABLED: Always return False - chemistry detection disabled
return False
# OLD CODE (kept for reference):
# from ..data.chemistry_data import CHEMISTRY_VARIABLES
# # In chemistry context, check if it's a known variable
# if in_chemistry_context:
# return text in CHEMISTRY_VARIABLES
# # Not in chemistry context, not a chemistry variable
# return False

380
scripts/omml_mathml2latex/utils/color_helpers.py Normal file
View File

@@ -0,0 +1,380 @@
"""
Color handling utilities for MathML to LaTeX conversion.
This module provides color optimization utilities including:
- Uniform color detection across entire equations
- Hex to LaTeX color conversion (named colors and RGB mode)
- Color suppression for optimization
"""
from lxml import etree as ET
from typing import Tuple, Set
def detect_uniform_color(node: ET.Element) -> str:
"""
Detect if entire equation subtree uses a single uniform color.
Args:
node: Root MathML element (typically <math> or <mrow>)
Returns:
Hex color string (e.g., '#FF0000') if uniform color detected, empty string otherwise.
Algorithm:
1. Recursively check all content-bearing elements (mi, mn, mo, mtext, etc.)
2. For each element, determine its effective color (from element or ancestors)
3. Return color only if ALL content elements have the SAME color
4. If ANY element has no color (default), it's not uniform
Examples:
>>> from lxml import etree as ET
>>> # All elements have same color
>>> xml = '<math><mstyle mathcolor="#FF0000"><mi>x</mi><mo>+</mo><mi>y</mi></mstyle></math>'
>>> tree = ET.fromstring(xml.encode('utf-8'))
>>> detect_uniform_color(tree)
'#FF0000'
>>> # Partial coloring (NOT uniform)
>>> xml = '<math><mi>x</mi><mstyle mathcolor="#FF0000"><mi>y</mi></mstyle></math>'
>>> tree = ET.fromstring(xml.encode('utf-8'))
>>> detect_uniform_color(tree)
''
>>> # Mixed colors
>>> xml = '<math><mi mathcolor="#FF0000">x</mi><mo mathcolor="#0000FF">+</mo></math>'
>>> tree = ET.fromstring(xml.encode('utf-8'))
>>> detect_uniform_color(tree)
''
"""
colors_found: Set[str] = set()
has_elements_without_color = False
# Content-bearing elements that should have consistent color
# These are the "leaf" elements that actually render visible content
CONTENT_ELEMENTS = {'mi', 'mn', 'mo', 'mtext', 'ms', 'merror'}
def get_local_name(tag: str) -> str:
"""Extract local tag name from potentially namespaced tag."""
return tag.split('}')[-1] if '}' in tag else tag
def get_effective_color(elem: ET.Element) -> str:
"""
Get the effective color for an element (from element or ancestors).
Returns empty string if no color is set.
"""
# Check element itself first
color = elem.get('mathcolor')
if color:
return color.upper()
# Check ancestors (walk up the tree)
parent = elem.getparent()
while parent is not None:
color = parent.get('mathcolor')
if color:
return color.upper()
parent = parent.getparent()
# No color found in element or ancestors
return ""
def check_colors(elem: ET.Element) -> None:
"""Recursively check colors of all content elements."""
nonlocal has_elements_without_color
tag = get_local_name(elem.tag)
# If this is a content-bearing element, check its effective color
if tag in CONTENT_ELEMENTS:
effective_color = get_effective_color(elem)
if effective_color:
colors_found.add(effective_color)
else:
# Found a content element without any color
has_elements_without_color = True
# Recurse into children
for child in elem:
check_colors(child)
check_colors(node)
# Uniform color only if:
# 1. We found exactly one unique color
# 2. ALL content elements have that color (none are without color)
if len(colors_found) == 1 and not has_elements_without_color:
return list(colors_found)[0]
# Not uniform: either no colors, multiple colors, or partial coloring
return ""
def hex_to_latex_color(hex_color: str) -> Tuple[str, str]:
"""
Convert hex color to LaTeX color name or RGB definition.
Args:
hex_color: Hex color string like '#FF0000' or 'FF0000'
Returns:
Tuple of (color_model, color_spec) where:
- color_model is '' for named colors or '[RGB]' for RGB mode
- color_spec is 'red' for named colors or '255,0,0' for RGB
Examples:
>>> hex_to_latex_color('#FF0000')
('', 'red')
>>> hex_to_latex_color('#E54C5E') # Custom color
('[RGB]', '229,76,94')
>>> hex_to_latex_color('000000') # Black (no #)
('', 'black')
Common Color Mappings:
#FF0000 → red
#00FF00 → green
#0000FF → blue
#000000 → black
#FFFFFF → white
#FFFF00 → yellow
#FF00FF → magenta
#00FFFF → cyan
"""
# Remove '#' if present
hex_color = hex_color.lstrip('#')
# Common color mappings (full hex codes only)
common_colors = {
'ff0000': 'red',
'00ff00': 'green',
'0000ff': 'blue',
'000000': 'black',
'ffffff': 'white',
'ffff00': 'yellow',
'ff00ff': 'magenta',
'00ffff': 'cyan',
}
hex_lower = hex_color.lower()
# Check for exact match in common colors
if hex_lower in common_colors:
return ('', common_colors[hex_lower])
# Convert to RGB mode for non-standard colors
try:
# Parse hex digits (supports 6-digit format)
if len(hex_color) == 6:
r = int(hex_color[0:2], 16)
g = int(hex_color[2:4], 16)
b = int(hex_color[4:6], 16)
return ('[RGB]', f'{r},{g},{b}')
else:
# Invalid format - default to black
return ('', 'black')
except (ValueError, IndexError):
# Parsing error - default to black
return ('', 'black')
def should_suppress_color(context) -> bool:
"""
Check if leaf-level colors should be suppressed.
Args:
context: ConversionContext object or converter with color optimization state
Returns:
True if leaf colors should be suppressed (uniform color detected), False otherwise
Usage:
This is used by visitors to check whether they should skip adding
\\textcolor wrappers when the entire equation has a uniform color.
"""
# Check for both attribute names (with and without underscore)
# Legacy adapter uses _suppress_leaf_colors (private), context uses suppress_leaf_colors (public)
return getattr(context, '_suppress_leaf_colors', False) or getattr(context, 'suppress_leaf_colors', False)
def wrap_with_uniform_color(latex: str, uniform_color: str) -> str:
"""
Wrap entire LaTeX equation with uniform color.
Args:
latex: The LaTeX equation string
uniform_color: Hex color string (e.g., '#FF0000')
Returns:
LaTeX wrapped with \\textcolor[RGB]{r,g,b}{...} or original if black
Examples:
>>> wrap_with_uniform_color('x^2', '#FF0000')
'\\\\textcolor{red}{x^2}'
>>> wrap_with_uniform_color('x^2', '#E54C5E')
'\\\\textcolor[RGB]{229,76,94}{x^2}'
>>> wrap_with_uniform_color('x^2', '#000000') # Black - no wrapping
'x^2'
"""
from ..utils import is_default_color
if not uniform_color or is_default_color(uniform_color):
# Black is default - no wrapping needed
return latex
# Convert to LaTeX color
color_model, color_spec = hex_to_latex_color(uniform_color)
# Skip if it's black after conversion
if color_spec == 'black':
return latex
# Wrap with appropriate format
if color_model:
# RGB mode
return f"\\textcolor{color_model}{{{color_spec}}}{{{latex}}}"
else:
# Named color
return f"\\textcolor{{{color_spec}}}{{{latex}}}"
# Mapping from Word highlight color names to CSS-compatible color values
# Word uses some non-standard names (e.g., darkyellow) that need conversion
WORD_HIGHLIGHT_TO_CSS = {
'yellow': 'yellow',
'green': 'lime', # Word "green" is bright green (lime in CSS)
'cyan': 'cyan',
'magenta': 'magenta',
'blue': 'blue',
'red': 'red',
'darkblue': 'darkblue',
'darkcyan': 'darkcyan',
'darkgreen': 'darkgreen',
'darkmagenta': 'darkmagenta',
'darkred': 'darkred',
'darkyellow': '#cccc00', # CSS has no "darkyellow"
'darkgray': 'darkgray',
'lightgray': 'lightgray',
'black': 'black',
'white': 'white',
}
def _highlight_color_to_css(color: str) -> str:
"""Convert a Word highlight color name or hex code to a CSS-compatible value.
Args:
color: Color name (e.g., 'yellow') or hex string (e.g., '#FFFF00')
Returns:
CSS-compatible color string
"""
if not color:
return ''
color_lower = color.lower().strip()
# If it's a hex color, return as-is
if color_lower.startswith('#') or (
len(color_lower) == 6 and all(c in '0123456789abcdef' for c in color_lower)
):
return f'#{color_lower.lstrip("#")}'
# Look up in Word highlight mapping
if color_lower in WORD_HIGHLIGHT_TO_CSS:
return WORD_HIGHLIGHT_TO_CSS[color_lower]
# Unknown named color - return as-is (CSS may still recognize it)
return color_lower
def wrap_with_background_color(latex: str, background_color: str) -> str:
"""
Wrap entire LaTeX equation with background color using \\bbox.
Uses MathJax \\bbox command to apply background color to the equation.
The \\bbox[<options>]{<content>} command renders a colored box around
the equation content.
Args:
latex: The LaTeX equation string
background_color: Color name (e.g., 'yellow', 'green') or hex string
Returns:
LaTeX wrapped with \\bbox[color]{content}, or original if no color
Examples:
>>> wrap_with_background_color('x^2', 'yellow')
'\\\\bbox[yellow]{x^2}'
>>> wrap_with_background_color('x^2', '#FFFF00')
'\\\\bbox[#ffff00]{x^2}'
>>> wrap_with_background_color('x^2', '') # No color - no wrapping
'x^2'
"""
if not background_color:
return latex
# Convert Word highlight color to CSS-compatible value
css_color = _highlight_color_to_css(background_color)
if not css_color:
return latex
# Wrap with \bbox for MathJax background color rendering
return f"\\bbox[{css_color}]{{{latex}}}"
def wrap_element_with_color(latex: str, node: ET.Element, context=None) -> str:
"""
Wrap element with color if present and not suppressed.
Args:
latex: The LaTeX string to wrap
node: MathML element that may have mathcolor attribute
context: Optional ConversionContext for suppression check
Returns:
Original LaTeX or wrapped with \\textcolor{color}{...}
This function respects the suppress_leaf_colors flag in the context.
If suppression is active, it skips individual element coloring.
Examples:
>>> from lxml import etree as ET
>>> node = ET.fromstring(b'<mi mathcolor="#FF0000">x</mi>')
>>> wrap_element_with_color('x', node)
'\\\\textcolor{red}{x}'
>>> # With suppression
>>> class Context:
... suppress_leaf_colors = True
>>> wrap_element_with_color('x', node, Context())
'x'
"""
from ..utils import is_default_color
# Check if suppression is active
if context and should_suppress_color(context):
return latex
# Get mathcolor attribute
mathcolor = node.get('mathcolor')
if not mathcolor or is_default_color(mathcolor):
return latex
# Convert and wrap
color_model, color_spec = hex_to_latex_color(mathcolor)
# Skip black (default color)
if color_spec == 'black':
return latex
# Wrap with appropriate format
if color_model:
return f"\\textcolor{color_model}{{{color_spec}}}{{{latex}}}"
else:
return f"\\textcolor{{{color_spec}}}{{{latex}}}"

55
scripts/omml_mathml2latex/utils/debug.py Normal file
View File

@@ -0,0 +1,55 @@
"""
Debug output control utility.
This module provides a centralized way to control debug output across the project.
Debug output is disabled by default and can be enabled via the OMML2TEX_DEBUG
environment variable.
Usage:
from omml2tex.utils.debug import debug_print
debug_print("[DEBUG] Some debug info") # Only prints if OMML2TEX_DEBUG=1
Environment Variables:
OMML2TEX_DEBUG: Set to "1" or "true" to enable debug output (default: disabled)
"""
import os
import sys
def is_debug_enabled() -> bool:
"""
Check if debug mode is enabled via environment variable.
Returns:
True if OMML2TEX_DEBUG is set to "1" or "true", False otherwise
"""
debug_value = os.environ.get('OMML2TEX_DEBUG', '').lower()
return debug_value in ('1', 'true', 'yes', 'on')
def debug_print(*args, **kwargs):
"""
Print debug output only if debug mode is enabled.
This function has the same signature as the built-in print() function,
but only outputs to stderr when OMML2TEX_DEBUG environment variable is set.
Args:
*args: Arguments to print
**kwargs: Keyword arguments for print (e.g., file, sep, end)
Examples:
>>> debug_print("[DEBUG] Processing equation")
>>> debug_print("[DEBUG] Value:", value, "Result:", result)
>>> debug_print("[DEBUG] Error", file=sys.stderr) # file=sys.stderr is default
"""
if is_debug_enabled():
# Default to stderr if file not specified
if 'file' not in kwargs:
kwargs['file'] = sys.stderr
print(*args, **kwargs)
__all__ = ['is_debug_enabled', 'debug_print']

136
scripts/omml_mathml2latex/utils/helpers.py Normal file
View File

@@ -0,0 +1,136 @@
"""
Pattern and symbol replacement helper functions.
This module provides utilities for applying symbol mappings and regex patterns
to text, used throughout the conversion pipeline.
"""
import re
import logging
logger = logging.getLogger(__name__)
def apply_symbol_replacements(text: str, mapping: dict) -> str:
"""Apply simple symbol replacements from a mapping dictionary.
Args:
text: Input text
mapping: Dictionary of {symbol: replacement}
Returns:
Text with all symbols replaced
Example:
>>> mapping = {'α': '\\\\alpha', 'β': '\\\\beta'}
>>> apply_symbol_replacements('α + β', mapping)
'\\\\alpha + \\\\beta'
"""
for symbol, replacement in mapping.items():
text = text.replace(symbol, replacement)
return text
def apply_pattern_replacements(text: str, patterns: list, verbose: bool = False) -> str:
"""Apply regex pattern replacements from a list of (pattern, replacement, description) tuples.
Args:
text: Input text
patterns: List of (pattern, replacement, description) tuples
verbose: If True, log each replacement
Returns:
Text with all patterns applied
Example:
>>> patterns = [(r' {3,}', ' ', 'Reduce multiple spaces')]
>>> apply_pattern_replacements('a b', patterns)
'a b'
"""
for pattern, replacement, description in patterns:
if verbose:
count = len(re.findall(pattern, text))
if count > 0:
logger.debug(f"Applying: {description} ({count} matches)")
text = re.sub(pattern, replacement, text)
return text
def apply_unicode_symbol_mapping(
text: str,
unicode_to_latex: dict,
additional_replacements: dict = None,
subscript_context: bool = False
) -> str:
"""Apply Unicode to LaTeX symbol mapping.
Args:
text: Input text containing Unicode symbols
unicode_to_latex: Main Unicode to LaTeX mapping dictionary
additional_replacements: Optional additional symbol replacements
subscript_context: If True, use subscript-specific mappings (with spaces)
Returns:
Text with Unicode symbols converted to LaTeX
Example:
>>> mapping = {'α': '\\\\alpha', '→': '\\\\to'}
>>> apply_unicode_symbol_mapping('α → β', mapping, {})
'\\\\alpha \\\\to β'
"""
# First apply standard UNICODE_TO_LATEX mapping
for symbol, latex in unicode_to_latex.items():
text = text.replace(symbol, latex)
# In subscript context, apply additional replacements with proper spacing
if subscript_context and additional_replacements:
for symbol, latex in additional_replacements.items():
text = text.replace(symbol, latex)
return text
def is_default_color(color: str) -> bool:
"""Check if a color value represents the default (black) color.
Args:
color: Color value in various formats (hex, name, rgb)
Returns:
True if the color is black/default, False otherwise
Examples:
>>> is_default_color('#000000')
True
>>> is_default_color('#000')
True
>>> is_default_color('black')
True
>>> is_default_color('rgb(0, 0, 0)')
True
>>> is_default_color('#FF0000')
False
>>> is_default_color('red')
False
"""
if not color:
return False
# Normalize the color string
color_lower = color.lower().strip()
# Check for common black color representations
black_colors = {
'#000000', # Full hex
'#000', # Short hex
'black', # Named color
'rgb(0,0,0)', # RGB without spaces
'rgb(0, 0, 0)', # RGB with spaces
'rgba(0,0,0,1)', # RGBA without spaces
'rgba(0, 0, 0, 1)', # RGBA with spaces
}
# Remove all whitespace for comparison
color_normalized = ''.join(color_lower.split())
return color_normalized in {c.replace(' ', '') for c in black_colors}

1009
scripts/omml_mathml2latex/utils/styling_postprocessor.py Normal file
View File

File diff suppressed because it is too large Load Diff

185
scripts/omml_mathml2latex/utils/text_processing.py Normal file
View File

@@ -0,0 +1,185 @@
"""
Text processing utilities for file I/O and LaTeX formatting.
This module provides functions for reading/writing text files and converting
various space characters to appropriate LaTeX spacing commands.
"""
import os
def read_text(path: str, encoding: str = "utf-8") -> str:
"""Read text from a file.
Args:
path: Path to the file to read
encoding: Text encoding (default: utf-8)
Returns:
File contents as string
Example:
>>> content = read_text('input.tex')
"""
with open(path, "r", encoding=encoding) as f:
return f.read()
def write_text(path: str, content: str, encoding: str = "utf-8") -> None:
"""Write text to a file, creating parent directories if needed.
Args:
path: Path to the file to write
content: Text content to write
encoding: Text encoding (default: utf-8)
Example:
>>> write_text('output.tex', '\\\\sin x')
"""
dirpath = os.path.dirname(path)
if dirpath:
os.makedirs(dirpath, exist_ok=True)
with open(path, "w", encoding=encoding) as f:
f.write(content)
def convert_spaces_to_latex(text: str) -> str:
"""Convert space characters in text to appropriate LaTeX spacing commands.
Handles:
- U+00A0 (NBSP, \\xa0) → ~ (tilde, non-breaking space in LaTeX)
- U+0020 (regular space) → \\\\ (backslash-space) for each space
Args:
text: Input text possibly containing various space characters
Returns:
Text with spaces converted to LaTeX commands
Example:
>>> convert_spaces_to_latex('sin x')
'sin\\\\ x'
>>> convert_spaces_to_latex('sin\\xa0x') # NBSP
'sin~x'
"""
if not text:
return text
# Quick check: if no spaces present, return as-is
if '\xa0' not in text and ' ' not in text:
return text
result = []
i = 0
while i < len(text):
char = text[i]
if char == '\xa0': # U+00A0: Non-breaking space
# Always use tilde for NBSP in LaTeX
result.append('~')
i += 1
elif char == ' ': # U+0020: Regular space
# Count consecutive spaces
space_count = 1
while i + space_count < len(text) and text[i + space_count] == ' ':
space_count += 1
# Convert each space to backslash-space (backslash followed by space)
# For N spaces, produce "\\ " repeated N times: "\\ \\ \\ "
# Each '\\ ' is already a complete unit, so use empty join (not space join)
result.append('\\ ' * space_count)
i += space_count # Skip all processed spaces
else:
# Regular character
result.append(char)
i += 1
# Join results but preserve trailing spaces that are part of LaTeX commands
result_str = ''.join(result)
# Don't strip if the string ends with LaTeX space command '\\ '
# The trailing space in '\\ ' is semantically significant
if result_str.endswith('\\ '):
return result_str
else:
return result_str.rstrip() # Remove trailing space from last element
def escape_latex_special_chars(text: str, preserve_nbsp: bool = True) -> str:
"""Escape special characters that have meaning in LaTeX.
LaTeX special characters that need escaping:
- $ (math mode delimiter)
- ^ (superscript)
- _ (subscript)
- # (parameter token)
- @ (letter category code)
- ! (negative thin space)
- % (comment)
- & (alignment tab)
- * (discretionary multiplication)
- ~ (tilde) → \\sim in math mode
Args:
text: Input text possibly containing special characters
preserve_nbsp: If True, preserve NBSP (\\xa0) for later conversion to ~
If False, convert NBSP to ~ before escaping
Returns:
Text with special characters escaped
Example:
>>> escape_latex_special_chars('$100 ^ 2')
'\\$100 \\^ 2'
>>> escape_latex_special_chars('H_2O')
'H\\_2O'
>>> escape_latex_special_chars('a ~ b')
'a \\sim b'
"""
if not text:
return text
# CRITICAL: Preserve marker strings used for brace resolution
# These markers distinguish source braces from generated LaTeX braces
if text in ('__LBRACE__', '__RBRACE__', '__DELIM__'):
return text
# Use a temporary placeholder for NBSP to preserve it during escaping
# This allows NBSP to be converted to ~ later for LaTeX spacing
NBSP_PLACEHOLDER = '\x00NBSP\x00' # Using null bytes as unlikely to appear in text
if preserve_nbsp and '\xa0' in text:
# Temporarily replace NBSP with placeholder
text = text.replace('\xa0', NBSP_PLACEHOLDER)
# Mapping of special characters to their escaped versions
# NOTE: In math mode, * is a valid operator and doesn't need escaping
# NOTE: { and } are NOT escaped here - they're handled via __LBRACE__/__RBRACE__ markers
# to distinguish source braces from generated LaTeX braces (e.g., \mathrm{...})
ESCAPE_MAP = {
'$': r'\$',
'%': r'\%',
'#': r'\#',
'&': r'\&',
'_': r'\_',
'^': r'\^',
# NOTE: '~' is NOT escaped here - it's handled via UNICODE_TO_LATEX mapping to \sim
'@': r'\@',
'!': r'\!',
# NOTE: '*' is NOT escaped - it's a valid multiplication operator in math mode
'{': r'\{',
'}': r'\}',
}
result = text
for char, escaped in ESCAPE_MAP.items():
result = result.replace(char, escaped)
# Restore NBSP placeholder back to NBSP for later processing
if preserve_nbsp and NBSP_PLACEHOLDER in result:
result = result.replace(NBSP_PLACEHOLDER, '\xa0')
return result

376
scripts/omml_mathml2latex/utils/timer.py Normal file
View File

@@ -0,0 +1,376 @@
"""
Performance timing utilities for omml2tex.
Provides comprehensive timing instrumentation with minimal overhead:
- Execution time tracking for critical operations
- Memory usage monitoring
- Hierarchical timing for nested operations
- Statistical summaries and reports
"""
import time
import functools
from typing import Dict, List, Optional, Callable, Any
from dataclasses import dataclass, field
from contextlib import contextmanager
@dataclass
class TimingRecord:
"""Single timing measurement record."""
name: str
start_time: float
end_time: Optional[float] = None
duration: Optional[float] = None
metadata: Dict[str, Any] = field(default_factory=dict)
parent: Optional[str] = None
def finish(self) -> None:
"""Mark record as finished and calculate duration."""
if self.end_time is None:
self.end_time = time.perf_counter()
self.duration = self.end_time - self.start_time
def is_finished(self) -> bool:
"""Check if timing record is complete."""
return self.end_time is not None
@dataclass
class TimingSummary:
"""Statistical summary of timing measurements."""
name: str
count: int
total_time: float
avg_time: float
min_time: float
max_time: float
metadata: Dict[str, Any] = field(default_factory=dict)
class PerformanceTimer:
"""
High-precision performance timer with hierarchical timing support.
Features:
- Sub-millisecond precision using time.perf_counter()
- Hierarchical timing for nested operations
- Minimal overhead (~1-2 microseconds per timing)
- Thread-safe operation tracking
- Statistical summaries and reporting
Example:
timer = PerformanceTimer()
# Context manager usage
with timer.measure("conversion"):
result = convert_mathml(xml)
# Decorator usage
@timer.time_function
def process_equation(xml):
return convert(xml)
# Manual timing
timer.start("parsing")
data = parse_xml(xml)
timer.stop("parsing")
# Get results
print(timer.get_summary())
"""
def __init__(self, enabled: bool = True):
"""
Initialize performance timer.
Args:
enabled: If False, timing is disabled (zero overhead)
"""
self.enabled = enabled
self._records: Dict[str, List[TimingRecord]] = {}
self._active: Dict[str, TimingRecord] = {}
self._parent_stack: List[str] = []
def start(self, name: str, metadata: Optional[Dict[str, Any]] = None) -> None:
"""
Start timing an operation.
Args:
name: Operation name (used for grouping)
metadata: Optional metadata to attach to record
"""
if not self.enabled:
return
parent = self._parent_stack[-1] if self._parent_stack else None
record = TimingRecord(
name=name,
start_time=time.perf_counter(),
metadata=metadata or {},
parent=parent
)
self._active[name] = record
self._parent_stack.append(name)
def stop(self, name: str, metadata: Optional[Dict[str, Any]] = None) -> Optional[float]:
"""
Stop timing an operation.
Args:
name: Operation name (must match start() call)
metadata: Optional additional metadata
Returns:
Duration in seconds, or None if timing disabled/not started
"""
if not self.enabled:
return None
if name not in self._active:
return None
record = self._active.pop(name)
record.finish()
if metadata:
record.metadata.update(metadata)
if name not in self._records:
self._records[name] = []
self._records[name].append(record)
# Remove from parent stack
if self._parent_stack and self._parent_stack[-1] == name:
self._parent_stack.pop()
return record.duration
@contextmanager
def measure(self, name: str, metadata: Optional[Dict[str, Any]] = None):
"""
Context manager for timing a block of code.
Args:
name: Operation name
metadata: Optional metadata
Yields:
TimingRecord that will be populated with results
Example:
with timer.measure("parsing", {"file": "input.xml"}):
data = parse_file()
"""
if not self.enabled:
yield None
return
self.start(name, metadata)
try:
yield self._active.get(name)
finally:
self.stop(name)
def time_function(self, name: Optional[str] = None):
"""
Decorator for timing function execution.
Args:
name: Optional custom name (defaults to function name)
Example:
@timer.time_function
def convert_equation(xml):
return process(xml)
@timer.time_function("custom_name")
def process_data(data):
return transform(data)
"""
def decorator(func: Callable) -> Callable:
operation_name = name or func.__name__
@functools.wraps(func)
def wrapper(*args, **kwargs):
if not self.enabled:
return func(*args, **kwargs)
with self.measure(operation_name):
return func(*args, **kwargs)
return wrapper
return decorator
def get_records(self, name: Optional[str] = None) -> List[TimingRecord]:
"""
Get timing records.
Args:
name: Optional operation name filter
Returns:
List of timing records (all records if name=None)
"""
if name:
return self._records.get(name, [])
all_records = []
for records in self._records.values():
all_records.extend(records)
return all_records
def get_summary(self, name: Optional[str] = None) -> Dict[str, TimingSummary]:
"""
Get statistical summary of timing measurements.
Args:
name: Optional operation name filter
Returns:
Dictionary mapping operation names to summaries
"""
summaries = {}
records_dict = {name: self._records[name]} if name and name in self._records else self._records
for op_name, records in records_dict.items():
if not records:
continue
durations = [r.duration for r in records if r.duration is not None]
if not durations:
continue
summaries[op_name] = TimingSummary(
name=op_name,
count=len(durations),
total_time=sum(durations),
avg_time=sum(durations) / len(durations),
min_time=min(durations),
max_time=max(durations)
)
return summaries
def get_total_time(self) -> float:
"""Get total time across all operations."""
summaries = self.get_summary()
return sum(s.total_time for s in summaries.values())
def format_report(self, include_hierarchy: bool = True) -> str:
"""
Format timing results as human-readable report.
Args:
include_hierarchy: Include parent-child relationships
Returns:
Formatted report string
"""
summaries = self.get_summary()
if not summaries:
return "No timing data collected"
lines = [
"=" * 80,
"Performance Timing Report".center(80),
"=" * 80,
""
]
# Sort by total time (descending)
sorted_summaries = sorted(
summaries.values(),
key=lambda s: s.total_time,
reverse=True
)
# Table header
lines.append(f"{'Operation':<30} {'Count':>8} {'Total':>12} {'Avg':>12} {'Min':>12} {'Max':>12}")
lines.append("-" * 80)
# Table rows
for summary in sorted_summaries:
lines.append(
f"{summary.name:<30} "
f"{summary.count:>8} "
f"{self._format_time(summary.total_time):>12} "
f"{self._format_time(summary.avg_time):>12} "
f"{self._format_time(summary.min_time):>12} "
f"{self._format_time(summary.max_time):>12}"
)
lines.append("-" * 80)
lines.append(f"{'TOTAL':<30} {'':<8} {self._format_time(self.get_total_time()):>12}")
lines.append("=" * 80)
# Hierarchy view
if include_hierarchy:
lines.append("")
lines.append("Hierarchical View:")
lines.append("-" * 80)
self._add_hierarchy_lines(lines)
return "\n".join(lines)
def _add_hierarchy_lines(self, lines: List[str], parent: Optional[str] = None, indent: int = 0):
"""Add hierarchical timing view to report."""
for name, records in self._records.items():
# Find records with matching parent
matching_records = [r for r in records if r.parent == parent]
if not matching_records:
continue
avg_time = sum(r.duration for r in matching_records if r.duration) / len(matching_records)
prefix = " " * indent + "└─ " if indent > 0 else ""
lines.append(f"{prefix}{name}: {self._format_time(avg_time)} avg ({len(matching_records)} calls)")
# Recursively add children
self._add_hierarchy_lines(lines, name, indent + 1)
@staticmethod
def _format_time(seconds: float) -> str:
"""Format time duration in human-readable form."""
if seconds < 1e-6:
return f"{seconds * 1e9:.2f} ns"
elif seconds < 1e-3:
return f"{seconds * 1e6:.2f} µs"
elif seconds < 1:
return f"{seconds * 1e3:.2f} ms"
else:
return f"{seconds:.2f} s"
def reset(self) -> None:
"""Clear all timing data."""
self._records.clear()
self._active.clear()
self._parent_stack.clear()
def enable(self) -> None:
"""Enable timing."""
self.enabled = True
def disable(self) -> None:
"""Disable timing."""
self.enabled = False
# Global timer instance for convenience
_global_timer = PerformanceTimer(enabled=False)
def get_global_timer() -> PerformanceTimer:
"""Get global timer instance."""
return _global_timer
def enable_global_timer() -> None:
"""Enable global timer."""
_global_timer.enable()
def disable_global_timer() -> None:
"""Disable global timer."""
_global_timer.disable()

205
scripts/omml_mathml2latex/utils/xml_utils.py Normal file
View File

@@ -0,0 +1,205 @@
"""
XML processing utilities for MathML and OMML handling.
This module provides helper functions for working with XML/MathML/OMML documents,
including namespace handling, text extraction, and HTML entity decoding.
"""
import re
import html
from typing import List
from lxml import etree as ET
def local_name(tag: str) -> str:
"""Extract local tag name without namespace.
Args:
tag: XML tag name, possibly with namespace prefix like "{http://...}tagname"
Returns:
Local tag name without namespace
Example:
>>> local_name("{http://www.w3.org/1998/Math/MathML}mi")
'mi'
>>> local_name("mi")
'mi'
"""
if "}" in tag:
return tag.split("}", 1)[1]
return tag
def text_content(node: ET._Element) -> str:
"""Flatten text under a node (best-effort).
NOTE: Does NOT strip() to preserve space-only content like NBSP and regular spaces.
Spaces are semantically meaningful in mathematical notation.
However, DOES remove invisible formatting characters like function application (\u2061)
and normalizes whitespace (converts newlines/tabs to single spaces).
Args:
node: XML element to extract text from
Returns:
Concatenated text content from node and all its children
Example:
>>> from lxml import etree as ET
>>> node = ET.fromstring('<mi>sin</mi>')
>>> text_content(node)
'sin'
"""
texts: List[str] = []
if node.text:
texts.append(node.text)
for child in node:
texts.append(text_content(child))
if child.tail:
texts.append(child.tail)
result = "".join(texts)
# Remove invisible formatting characters that interfere with conversion
# but preserve meaningful spaces (regular space, nbsp)
invisible_chars = [
'\u2061', # FUNCTION APPLICATION (causes sin⁡ → \mathrm{sin})
'\u2062', # INVISIBLE TIMES
'\u2063', # INVISIBLE SEPARATOR
'\u2064', # INVISIBLE PLUS
'\uFEFF', # ZERO WIDTH NO-BREAK SPACE
'\u200B', # ZERO WIDTH SPACE
'\u200C', # ZERO WIDTH NON-JOINER
'\u200D', # ZERO WIDTH JOINER
]
for char in invisible_chars:
result = result.replace(char, '')
# Normalize whitespace: Remove newlines and tabs completely
# This fixes both multiline fractions like \dfrac{\n 2\n}
# AND prevents formatting whitespace from JSON/XML from being converted to LaTeX spacing
# Preserve NBSP (\xa0) as it's semantically meaningful
result = re.sub(r'[\n\r\t]+', '', result)
# Collapse multiple consecutive regular spaces into one
# This prevents " 2 " → should be " 2 " or just "2"
# But don't collapse NBSP (\xa0) as it's semantically meaningful
result = re.sub(r' +', ' ', result)
# Strip leading/trailing spaces that come from XML formatting
# These are not semantically meaningful - they're just from pretty-printed XML
# True semantic spaces (like in operators) are preserved within the content
result = result.strip()
return result
def decode_html_entities(text: str) -> str:
"""Decode HTML entities in text, particularly &quot; from web page copies.
IMPORTANT: This function is designed for HTML content, NOT for XML/MathML!
DO NOT use this on XML/MathML strings before XML parsing, as it will break
valid XML numeric character references like &#x003C; which XML parsers
handle natively.
Args:
text: Text possibly containing HTML entities
Returns:
Text with HTML named entities decoded (numeric entities preserved for XML)
Example:
>>> decode_html_entities('&lt;math&gt; &amp; &quot;test&quot;')
'<math> & "test"'
>>> decode_html_entities('<mo>&#x003C;</mo>') # Preserves numeric entities
'<mo>&#x003C;</mo>'
"""
# CRITICAL FIX: Do NOT use html.unescape() on XML/MathML content!
# html.unescape() decodes numeric character references like &#x003C; to <
# which breaks XML parsing when < is inside element content.
#
# Instead, only decode common HTML named entities that might appear
# in web page copies, and leave numeric entities for XML parser to handle.
# Handle specific named entities that might appear in HTML copies
# but preserve numeric entities (&#x...;, &#...;) for XML parsing
entity_map = {
'&quot;': '"',
'&amp;': '&',
'&lt;': '<',
'&gt;': '>',
'&apos;': "'",
'&nbsp;': '\xa0', # Non-breaking space
}
for entity, char in entity_map.items():
text = text.replace(entity, char)
return text
def extract_math_equations(text: str) -> List[str]:
"""Extract multiple MathML equations from text starting with <math tag.
Args:
text: Text possibly containing one or more MathML equations
Returns:
List of MathML equation strings (empty list if none found)
Example:
>>> text = '<math><mi>x</mi></math> and <math><mi>y</mi></math>'
>>> equations = extract_math_equations(text)
>>> len(equations)
2
"""
# CRITICAL FIX: Do NOT decode HTML entities before regex extraction!
# The input is XML/MathML which contains numeric character references
# like &#x003C; that are valid XML and must be preserved.
# XML parsers (lxml) will handle entity decoding automatically.
# Decoding them here would break valid XML like <mo>&#x003C;</mo> → <mo><</mo>
#
# Original line (REMOVED): text = decode_html_entities(text)
equations = []
# Find all <math> tags with their content
# Use a more robust regex that handles nested tags and various formatting
pattern = r'<math[^>]*>.*?</math>'
matches = re.findall(pattern, text, re.DOTALL | re.IGNORECASE)
for match in matches:
# Clean up the equation
equation = match.strip()
if equation:
# Decode HTML entities in the extracted equation
# This handles cases like &quot; → " which are needed for valid XML
equation = decode_html_entities(equation)
equations.append(equation)
# If no <math> tags found, check if the entire input is a single equation
if not equations and text.strip().startswith('<math'):
# Try to find the end of the math tag
text = text.strip()
# Simple case: if it looks like a complete math equation
if '</math>' in text.lower():
# Decode HTML entities before appending
text = decode_html_entities(text)
equations.append(text)
else:
# Try to fix incomplete equation by adding closing tag
if not text.lower().endswith('</math>'):
text += '</math>'
# Decode HTML entities before appending
text = decode_html_entities(text)
equations.append(text)
return equations
# Alias for backward compatibility (some code uses _local, some uses local_name)
_local = local_name
_text_content = text_content