Quentin/DoLuEdu-WordFileParse
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

init: WordFileParse docx→HTML 解析服务（含 Word EQ 域公式修复）

Browse Source 
服务：Spring Boot 3.2 + docx4j 8.3.9，POST /api/v3/server_office_file/file/parse
      接受 .docx，返回 HTML（公式 → LaTeX，图片 → RustFS 外链）

公式支持三类来源：
- OMML（Word 2007+ 公式）      → Python omml_mathml2latex
- MathType OLE 对象            → Ruby mathtype2mathml → MathML → LaTeX
- Word EQ 域（老教辅主力）      → EqToLatexUtil（本次新增）

本次改动：
1. 新增 java-svc/.../core/EqToLatexUtil.java：
   移植自 jby backend-services office-core，EQ 语法 → LaTeX
   （\f→\dfrac、\r→\sqrt、\i→\int/\sum/\prod、\s→^{}/_{}、\b→\left\right、
     \a→array、\x→\boxed/\overline、\o\al 堆叠、圈码、化学符号、极限、连等式）
   品牌标记 JBYU* → WPU*，与采集端一致
2. CtoMathFinder：接上 EQ 分支（原为 not-ported 桩），域代码经
   EqToLatexUtil.convert 转 LaTeX 写入 eqLatexMap（WPEQNUM 占位符）
3. WordParseUtil.dealPlaceholder：EQ 的 LaTeX 同样加 \( \) 包裹
   （原先因 WPEQNUM 例外不加包裹，前端/MathJax 不渲染）

修前症状：教辅 Word 里分式/方程组/上下标整段丢失（如 {−1/3, 1} 变成 {－，1}）
修后实测：46 份教辅 2908 处 EQ 域全部还原，公式总数 2954 处、零未包裹残留

安全：application.yml（含 RustFS 密钥）不入库，提供 application.yml.example；
      OssService 的 @Value 默认密钥已清空
This commit is contained in:
Quentin
2026-09-29 09:43:13 +08:00
commit 3271fee5cc
360 changed files with 86082 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

205
scripts/omml_mathml2latex/utils/xml_utils.py Normal file
View File

@@ -0,0 +1,205 @@
"""
XML processing utilities for MathML and OMML handling.
This module provides helper functions for working with XML/MathML/OMML documents,
including namespace handling, text extraction, and HTML entity decoding.
"""
import re
import html
from typing import List
from lxml import etree as ET
def local_name(tag: str) -> str:
"""Extract local tag name without namespace.
Args:
tag: XML tag name, possibly with namespace prefix like "{http://...}tagname"
Returns:
Local tag name without namespace
Example:
>>> local_name("{http://www.w3.org/1998/Math/MathML}mi")
'mi'
>>> local_name("mi")
'mi'
"""
if "}" in tag:
return tag.split("}", 1)[1]
return tag
def text_content(node: ET._Element) -> str:
"""Flatten text under a node (best-effort).
NOTE: Does NOT strip() to preserve space-only content like NBSP and regular spaces.
Spaces are semantically meaningful in mathematical notation.
However, DOES remove invisible formatting characters like function application (\u2061)
and normalizes whitespace (converts newlines/tabs to single spaces).
Args:
node: XML element to extract text from
Returns:
Concatenated text content from node and all its children
Example:
>>> from lxml import etree as ET
>>> node = ET.fromstring('<mi>sin</mi>')
>>> text_content(node)
'sin'
"""
texts: List[str] = []
if node.text:
texts.append(node.text)
for child in node:
texts.append(text_content(child))
if child.tail:
texts.append(child.tail)
result = "".join(texts)
# Remove invisible formatting characters that interfere with conversion
# but preserve meaningful spaces (regular space, nbsp)
invisible_chars = [
'\u2061', # FUNCTION APPLICATION (causes sin⁡ → \mathrm{sin})
'\u2062', # INVISIBLE TIMES
'\u2063', # INVISIBLE SEPARATOR
'\u2064', # INVISIBLE PLUS
'\uFEFF', # ZERO WIDTH NO-BREAK SPACE
'\u200B', # ZERO WIDTH SPACE
'\u200C', # ZERO WIDTH NON-JOINER
'\u200D', # ZERO WIDTH JOINER
]
for char in invisible_chars:
result = result.replace(char, '')
# Normalize whitespace: Remove newlines and tabs completely
# This fixes both multiline fractions like \dfrac{\n 2\n}
# AND prevents formatting whitespace from JSON/XML from being converted to LaTeX spacing
# Preserve NBSP (\xa0) as it's semantically meaningful
result = re.sub(r'[\n\r\t]+', '', result)
# Collapse multiple consecutive regular spaces into one
# This prevents " 2 " → should be " 2 " or just "2"
# But don't collapse NBSP (\xa0) as it's semantically meaningful
result = re.sub(r' +', ' ', result)
# Strip leading/trailing spaces that come from XML formatting
# These are not semantically meaningful - they're just from pretty-printed XML
# True semantic spaces (like in operators) are preserved within the content
result = result.strip()
return result
def decode_html_entities(text: str) -> str:
"""Decode HTML entities in text, particularly &quot; from web page copies.
IMPORTANT: This function is designed for HTML content, NOT for XML/MathML!
DO NOT use this on XML/MathML strings before XML parsing, as it will break
valid XML numeric character references like &#x003C; which XML parsers
handle natively.
Args:
text: Text possibly containing HTML entities
Returns:
Text with HTML named entities decoded (numeric entities preserved for XML)
Example:
>>> decode_html_entities('&lt;math&gt; &amp; &quot;test&quot;')
'<math> & "test"'
>>> decode_html_entities('<mo>&#x003C;</mo>') # Preserves numeric entities
'<mo>&#x003C;</mo>'
"""
# CRITICAL FIX: Do NOT use html.unescape() on XML/MathML content!
# html.unescape() decodes numeric character references like &#x003C; to <
# which breaks XML parsing when < is inside element content.
#
# Instead, only decode common HTML named entities that might appear
# in web page copies, and leave numeric entities for XML parser to handle.
# Handle specific named entities that might appear in HTML copies
# but preserve numeric entities (&#x...;, &#...;) for XML parsing
entity_map = {
'&quot;': '"',
'&amp;': '&',
'&lt;': '<',
'&gt;': '>',
'&apos;': "'",
'&nbsp;': '\xa0', # Non-breaking space
}
for entity, char in entity_map.items():
text = text.replace(entity, char)
return text
def extract_math_equations(text: str) -> List[str]:
"""Extract multiple MathML equations from text starting with <math tag.
Args:
text: Text possibly containing one or more MathML equations
Returns:
List of MathML equation strings (empty list if none found)
Example:
>>> text = '<math><mi>x</mi></math> and <math><mi>y</mi></math>'
>>> equations = extract_math_equations(text)
>>> len(equations)
2
"""
# CRITICAL FIX: Do NOT decode HTML entities before regex extraction!
# The input is XML/MathML which contains numeric character references
# like &#x003C; that are valid XML and must be preserved.
# XML parsers (lxml) will handle entity decoding automatically.
# Decoding them here would break valid XML like <mo>&#x003C;</mo> → <mo><</mo>
#
# Original line (REMOVED): text = decode_html_entities(text)
equations = []
# Find all <math> tags with their content
# Use a more robust regex that handles nested tags and various formatting
pattern = r'<math[^>]*>.*?</math>'
matches = re.findall(pattern, text, re.DOTALL | re.IGNORECASE)
for match in matches:
# Clean up the equation
equation = match.strip()
if equation:
# Decode HTML entities in the extracted equation
# This handles cases like &quot; → " which are needed for valid XML
equation = decode_html_entities(equation)
equations.append(equation)
# If no <math> tags found, check if the entire input is a single equation
if not equations and text.strip().startswith('<math'):
# Try to find the end of the math tag
text = text.strip()
# Simple case: if it looks like a complete math equation
if '</math>' in text.lower():
# Decode HTML entities before appending
text = decode_html_entities(text)
equations.append(text)
else:
# Try to fix incomplete equation by adding closing tag
if not text.lower().endswith('</math>'):
text += '</math>'
# Decode HTML entities before appending
text = decode_html_entities(text)
equations.append(text)
return equations
# Alias for backward compatibility (some code uses _local, some uses local_name)
_local = local_name
_text_content = text_content