| 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407 |
- <!DOCTYPE html>
- <html lang="zh-CN">
- <head>
- <meta charset="UTF-8">
- <meta name="viewport" content="width=device-width, initial-scale=1.0">
- <title>阿里云百炼模型下线监控系统 · 技术方案</title>
- <style>
- :root {
- --bg: #0b1020;
- --panel: #131a30;
- --panel2: #1a2340;
- --line: #2a3558;
- --text: #e6ecff;
- --muted: #93a0c4;
- --blue: #38bdf8;
- --cyan: #2dd4bf;
- --amber: #fbbf24;
- --red: #f87171;
- --violet: #a78bfa;
- }
- * { box-sizing: border-box; margin: 0; padding: 0; }
- body {
- font-family: "Microsoft YaHei", "PingFang SC", "Segoe UI", sans-serif;
- background: radial-gradient(1200px 600px at 20% -10%, #16224a 0%, var(--bg) 55%);
- color: var(--text);
- line-height: 1.7;
- padding: 32px 20px 80px;
- }
- .wrap { max-width: 1080px; margin: 0 auto; }
- header { text-align: center; padding: 26px 0 8px; }
- header h1 {
- font-size: 30px; letter-spacing: 1px;
- background: linear-gradient(90deg, var(--blue), var(--cyan));
- -webkit-background-clip: text; background-clip: text; color: transparent;
- }
- header .sub { color: var(--muted); margin-top: 10px; font-size: 15px; }
- .tags { margin-top: 14px; display: flex; gap: 10px; justify-content: center; flex-wrap: wrap; }
- .tag {
- border: 1px solid var(--line); background: var(--panel);
- padding: 4px 14px; border-radius: 999px; font-size: 13px; color: var(--muted);
- }
- .tag b { color: var(--blue); font-weight: 600; }
- section {
- background: var(--panel);
- border: 1px solid var(--line);
- border-radius: 16px;
- padding: 26px 30px;
- margin-top: 26px;
- }
- section h2 {
- font-size: 20px; margin-bottom: 6px;
- display: flex; align-items: center; gap: 10px;
- }
- section h2 .no {
- width: 28px; height: 28px; flex: 0 0 28px;
- border-radius: 8px; display: inline-flex; align-items: center; justify-content: center;
- font-size: 14px; font-weight: 700; color: #06121f;
- }
- .n1 { background: var(--blue); } .n2 { background: var(--cyan); }
- .n3 { background: var(--amber); } .n4 { background: var(--red); }
- .n5 { background: var(--violet); } .n6 { background: var(--blue); }
- .n7 { background: var(--cyan); }
- section .lead { color: var(--muted); font-size: 14px; margin-bottom: 18px; }
- .cards { display: grid; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); gap: 14px; }
- .card {
- background: var(--panel2); border: 1px solid var(--line);
- border-radius: 12px; padding: 16px 18px;
- }
- .card h4 { font-size: 15px; margin-bottom: 6px; }
- .card h4 i { font-style: normal; margin-right: 8px; }
- .card p { font-size: 13.5px; color: var(--muted); }
- .svgbox { margin-top: 14px; background: #0d1426; border: 1px solid var(--line); border-radius: 12px; padding: 8px; }
- .svgbox svg { width: 100%; height: auto; display: block; }
- .note {
- margin-top: 12px; font-size: 13px; color: var(--muted);
- border-left: 3px solid var(--amber); padding: 6px 12px; background: rgba(251,191,36,.06);
- }
- .note.blue { border-color: var(--blue); background: rgba(56,189,248,.06); }
- .grid2 { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }
- @media (max-width: 760px) { .grid2 { grid-template-columns: 1fr; } }
- .steps { list-style: none; }
- .steps li {
- padding: 10px 0 10px 34px; position: relative; font-size: 14px;
- border-bottom: 1px dashed var(--line);
- }
- .steps li:last-child { border-bottom: none; }
- .steps li::before {
- content: attr(data-i); position: absolute; left: 0; top: 11px;
- width: 22px; height: 22px; border-radius: 50%;
- background: var(--panel2); border: 1px solid var(--blue);
- display: flex; align-items: center; justify-content: center; font-size: 12px; color: var(--blue);
- }
- .steps li b { color: var(--blue); }
- footer { text-align: center; color: var(--muted); font-size: 12.5px; margin-top: 34px; }
- </style>
- </head>
- <body>
- <div class="wrap">
- <header>
- <h1>阿里云百炼 · 模型下线监控系统</h1>
- <div class="sub">面向模型批量下架 / API 失效风险的自动化监控与聚合告警方案</div>
- <div class="tags">
- <span class="tag"><b>Python</b> 技术栈</span>
- <span class="tag"><b>DashScope</b> 官方 SDK</span>
- <span class="tag">双通道监测</span>
- <span class="tag">零本地状态 · 云端去重</span>
- <span class="tag">钉钉 + 邮件双出口</span>
- </div>
- </header>
- <!-- ============ 1. 系统概览 ============ -->
- <section>
- <h2><span class="no n1">1</span>系统概览</h2>
- <p class="lead">应对"阿里云百炼不定期下架过期模型、API 用着用着突然失效"的问题,构建一个部署在 Linux 服务器上的轻量监控服务。</p>
- <div class="cards">
- <div class="card"><h4><i>🎯</i>核心目标</h4><p>第一时间感知模型下线与请求异常,把"用户被动发现 API 失效"变为"系统主动告警"。</p></div>
- <div class="card"><h4><i>🔁</i>双通道监测</h4><p>① 定时轮询云端邮箱,监听阿里云模型下线通知;② 按配置频次主动调用每个模型验证可用性。</p></div>
- <div class="card"><h4><i>📮</i>零本地状态</h4><p>不依赖任何本地文件做缓存;已告警标识直接查询<strong>云端发件箱</strong>获取,重启无副作用、多机可部署。</p></div>
- <div class="card"><h4><i>📣</i>聚合告警</h4><p>本轮所有异常合并为一条消息,同时推送<strong>钉钉群机器人</strong>与<strong>邮件群发</strong>,通知到公司内部群。</p></div>
- </div>
- </section>
- <!-- ============ 2. 系统架构 ============ -->
- <section>
- <h2><span class="no n2">2</span>系统架构</h2>
- <p class="lead">三层结构:外部依赖(左侧) → 监控系统核心(中) → 通知出口(下)。调度器按配置频次触发整轮流程。</p>
- <div class="svgbox">
- <svg viewBox="0 0 1000 640" xmlns="http://www.w3.org/2000/svg" font-family="'Microsoft YaHei','PingFang SC',sans-serif">
- <defs>
- <marker id="arr" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#5b6b96"/>
- </marker>
- <marker id="arrc" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#2dd4bf"/>
- </marker>
- <filter id="shadow" x="-20%" y="-20%" width="140%" height="140%">
- <feDropShadow dx="0" dy="4" stdDeviation="6" flood-color="#000" flood-opacity=".4"/>
- </filter>
- </defs>
- <!-- 外部系统 -->
- <g filter="url(#shadow)">
- <rect x="70" y="36" width="260" height="64" rx="12" fill="#16214a" stroke="#38bdf8" stroke-width="1.4"/>
- <text x="200" y="66" fill="#38bdf8" text-anchor="middle" font-size="15" font-weight="700">阿里云百炼 · DashScope API</text>
- <text x="200" y="86" fill="#93a0c4" text-anchor="middle" font-size="12">被监控对象(模型推理服务)</text>
- <rect x="670" y="36" width="260" height="64" rx="12" fill="#16214a" stroke="#fbbf24" stroke-width="1.4"/>
- <text x="800" y="66" fill="#fbbf24" text-anchor="middle" font-size="15" font-weight="700">企业邮箱(云端)</text>
- <text x="800" y="86" fill="#93a0c4" text-anchor="middle" font-size="12">收件箱 INBOX + 发件箱 Sent</text>
- </g>
- <!-- 监控系统容器 -->
- <rect x="60" y="150" width="880" height="330" rx="18" fill="#111a33" stroke="#2a3558" stroke-width="1.4" stroke-dasharray="6 5"/>
- <text x="500" y="176" fill="#5b6b96" text-anchor="middle" font-size="14" font-weight="700">监 控 系 统 · 零本地状态(无任何本地文件缓存)</text>
- <!-- 模块行 -->
- <g filter="url(#shadow)">
- <rect x="110" y="205" width="210" height="78" rx="12" fill="#1a2340" stroke="#38bdf8"/>
- <text x="215" y="234" fill="#e6ecff" text-anchor="middle" font-size="14" font-weight="700">定时调度器</text>
- <text x="215" y="256" fill="#93a0c4" text-anchor="middle" font-size="12">按配置频次触发整轮</text>
- <rect x="395" y="205" width="210" height="78" rx="12" fill="#1a2340" stroke="#fbbf24"/>
- <text x="500" y="234" fill="#e6ecff" text-anchor="middle" font-size="14" font-weight="700">邮件检查器</text>
- <text x="500" y="256" fill="#93a0c4" text-anchor="middle" font-size="12">云端收件监听 + 发件箱去重</text>
- <rect x="680" y="205" width="210" height="78" rx="12" fill="#1a2340" stroke="#2dd4bf"/>
- <text x="785" y="234" fill="#e6ecff" text-anchor="middle" font-size="14" font-weight="700">模型探测器</text>
- <text x="785" y="256" fill="#93a0c4" text-anchor="middle" font-size="12">chat / image / audio 三种类型</text>
- </g>
- <!-- 聚合器 -->
- <g filter="url(#shadow)">
- <rect x="300" y="330" width="400" height="78" rx="12" fill="#1a2340" stroke="#a78bfa"/>
- <text x="500" y="360" fill="#e6ecff" text-anchor="middle" font-size="14" font-weight="700">告警聚合器</text>
- <text x="500" y="382" fill="#93a0c4" text-anchor="middle" font-size="12">下线通知 / 模型异常 → 聚合为一条告警</text>
- </g>
- <!-- 出口 -->
- <g filter="url(#shadow)">
- <rect x="150" y="556" width="290" height="64" rx="12" fill="#16214a" stroke="#f87171"/>
- <text x="295" y="586" fill="#f87171" text-anchor="middle" font-size="14" font-weight="700">钉钉群机器人</text>
- <text x="295" y="606" fill="#93a0c4" text-anchor="middle" font-size="12">HMAC 加签 · Markdown 推送</text>
- <rect x="560" y="556" width="290" height="64" rx="12" fill="#16214a" stroke="#f87171"/>
- <text x="705" y="586" fill="#f87171" text-anchor="middle" font-size="14" font-weight="700">SMTP 群发 → 通知人邮箱</text>
- <text x="705" y="606" fill="#93a0c4" text-anchor="middle" font-size="12">标题带 [ALERT-MSG:ID],留存云端发件箱</text>
- </g>
- <!-- 箭头 -->
- <line x1="330" y1="84" x2="740" y2="200" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr)"/>
- <line x1="820" y1="100" x2="620" y2="200" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr)"/>
- <path d="M820,150 L820,120 L500,120 L500,200" fill="none" stroke="#2dd4bf" stroke-width="1.6" stroke-dasharray="5 4" marker-end="url(#arrc)"/>
- <text x="500" y="112" fill="#2dd4bf" text-anchor="middle" font-size="11">查询云端发件箱 Sent 去重</text>
- <line x1="215" y1="283" x2="215" y2="330" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr)"/>
- <line x1="500" y1="283" x2="500" y2="330" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr)"/>
- <line x1="785" y1="283" x2="700" y2="330" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr)"/>
- <path d="M380,408 L295,552" fill="none" stroke="#f87171" stroke-width="1.8" marker-end="url(#arr)"/>
- <path d="M620,408 L680,552" fill="none" stroke="#f87171" stroke-width="1.8" marker-end="url(#arr)"/>
- </svg>
- </div>
- <div class="note blue">关键设计:监控系统<strong>不落任何本地文件</strong>。去重依据来自<strong>云端发件箱留存</strong>——服务重启、迁移、多副本部署均不会丢失告警状态。</div>
- </section>
- <!-- ============ 3. 单轮执行流程 ============ -->
- <section>
- <h2><span class="no n3">3</span>单轮执行流程</h2>
- <p class="lead">每次轮询:邮件检查与模型探测<strong>并行</strong>执行;本轮结束时统一汇总,异常事件聚合推送,全部正常则静默。</p>
- <div class="svgbox">
- <svg viewBox="0 0 1000 700" xmlns="http://www.w3.org/2000/svg" font-family="'Microsoft YaHei','PingFang SC',sans-serif">
- <defs>
- <marker id="arr2" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#5b6b96"/>
- </marker>
- <marker id="arrr" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#f87171"/>
- </marker>
- <marker id="arrg" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#2dd4bf"/>
- </marker>
- </defs>
- <!-- 开始 -->
- <rect x="380" y="26" width="240" height="52" rx="26" fill="#16214a" stroke="#38bdf8" stroke-width="1.6"/>
- <text x="500" y="58" fill="#38bdf8" text-anchor="middle" font-size="14" font-weight="700">定时触发(按配置频次)</text>
- <line x1="500" y1="78" x2="500" y2="116" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr2)"/>
- <!-- 并行容器 -->
- <rect x="120" y="120" width="760" height="130" rx="14" fill="#0d1426" stroke="#2a3558" stroke-dasharray="6 5"/>
- <text x="500" y="146" fill="#5b6b96" text-anchor="middle" font-size="13" font-weight="700">并 行 执 行</text>
- <rect x="170" y="162" width="330" height="72" rx="12" fill="#1a2340" stroke="#fbbf24"/>
- <text x="335" y="188" fill="#e6ecff" text-anchor="middle" font-size="13" font-weight="700">检查云端收件箱(近 3 天)</text>
- <text x="335" y="210" fill="#93a0c4" text-anchor="middle" font-size="12">关键词匹配 · 发件人过滤 · 发件箱去重</text>
- <rect x="540" y="162" width="330" height="72" rx="12" fill="#1a2340" stroke="#2dd4bf"/>
- <text x="705" y="188" fill="#e6ecff" text-anchor="middle" font-size="13" font-weight="700">探测全部配置模型</text>
- <text x="705" y="210" fill="#93a0c4" text-anchor="middle" font-size="12">dashscope SDK · chat / image / audio</text>
- <line x1="335" y1="250" x2="480" y2="310" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr2)"/>
- <line x1="705" y1="250" x2="560" y2="310" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr2)"/>
- <!-- 判定菱形 -->
- <path d="M500,310 L650,378 L500,446 L350,378 Z" fill="#16214a" stroke="#fbbf24" stroke-width="1.6"/>
- <text x="500" y="378" fill="#fbbf24" text-anchor="middle" font-size="14" font-weight="700">存在下线通知</text>
- <text x="500" y="400" fill="#fbbf24" text-anchor="middle" font-size="14" font-weight="700">或模型异常?</text>
- <!-- 否 分支 -->
- <line x1="500" y1="446" x2="500" y2="500" stroke="#2dd4bf" stroke-width="1.6" marker-end="url(#arrg)"/>
- <rect x="390" y="500" width="220" height="50" rx="10" fill="#1a2340" stroke="#2dd4bf"/>
- <text x="500" y="530" fill="#2dd4bf" text-anchor="middle" font-size="13" font-weight="700">本轮结束 · 静默无操作</text>
- <!-- 是 分支 -->
- <line x1="650" y1="378" x2="760" y2="378" stroke="#f87171" stroke-width="1.6" marker-end="url(#arrr)"/>
- <text x="698" y="366" fill="#f87171" text-anchor="middle" font-size="12">是</text>
- <text x="320" y="366" fill="#2dd4bf" text-anchor="middle" font-size="12">否</text>
- <rect x="762" y="336" width="220" height="84" rx="12" fill="#1a2340" stroke="#f87171"/>
- <text x="872" y="364" fill="#e6ecff" text-anchor="middle" font-size="13" font-weight="700">聚合告警</text>
- <text x="872" y="386" fill="#93a0c4" text-anchor="middle" font-size="12">邮件事件逐条 + 模型异常合并</text>
- <line x1="872" y1="420" x2="872" y2="470" stroke="#f87171" stroke-width="1.6" marker-end="url(#arrr)"/>
- <rect x="762" y="472" width="220" height="84" rx="12" fill="#1a2340" stroke="#f87171"/>
- <text x="872" y="500" fill="#e6ecff" text-anchor="middle" font-size="13" font-weight="700">双出口推送</text>
- <text x="872" y="522" fill="#93a0c4" text-anchor="middle" font-size="12">钉钉 + SMTP 群发,标题带 [ALERT-MSG]</text>
- <line x1="872" y1="556" x2="872" y2="600" stroke="#f87171" stroke-width="1.6" marker-end="url(#arrr)"/>
- <rect x="762" y="602" width="220" height="50" rx="10" fill="#1a2340" stroke="#2dd4bf"/>
- <text x="872" y="632" fill="#2dd4bf" text-anchor="middle" font-size="13" font-weight="700">告警留存云端发件箱(供下轮去重)</text>
- </svg>
- </div>
- </section>
- <!-- ============ 4. 调度策略 ============ -->
- <section>
- <h2><span class="no n4">4</span>调度策略</h2>
- <p class="lead">配置"每天执行次数"即可,系统按 24 / N 小时均匀分布整点触发;也可用标准 cron 表达式精确指定时刻。</p>
- <div class="grid2">
- <div class="svgbox">
- <svg viewBox="0 0 520 150" xmlns="http://www.w3.org/2000/svg" font-family="'Microsoft YaHei','PingFang SC',sans-serif">
- <line x1="40" y1="90" x2="480" y2="90" stroke="#2a3558" stroke-width="2"/>
- <circle cx="60" cy="90" r="12" fill="#16214a" stroke="#f87171" stroke-width="2"/>
- <circle cx="300" cy="90" r="12" fill="#16214a" stroke="#38bdf8" stroke-width="2"/>
- <circle cx="460" cy="90" r="12" fill="#16214a" stroke="#2dd4bf" stroke-width="2"/>
- <text x="60" y="52" fill="#f87171" text-anchor="middle" font-size="15" font-weight="700">00:00</text>
- <text x="60" y="70" fill="#93a0c4" text-anchor="middle" font-size="11">触发点</text>
- <text x="300" y="52" fill="#38bdf8" text-anchor="middle" font-size="15" font-weight="700">12:00</text>
- <text x="300" y="70" fill="#93a0c4" text-anchor="middle" font-size="11">触发点</text>
- <text x="460" y="52" fill="#2dd4bf" text-anchor="middle" font-size="15" font-weight="700">24:00</text>
- <text x="460" y="70" fill="#93a0c4" text-anchor="middle" font-size="11">次日零点</text>
- <text x="260" y="128" fill="#93a0c4" text-anchor="middle" font-size="12">示例:每天 2 次 → 00:00 与 12:00</text>
- </svg>
- </div>
- <div>
- <ul class="steps">
- <li data-i="1"><b>数字频次</b>:配置 <code>daily_count = N</code>,即每天 N 次,按 24/N 小时均匀分布整点。</li>
- <li data-i="2"><b>精确时刻</b>:可选用标准 cron 表达式(分 时 日 月 周)指定任意时刻,优先级高于数字频次。</li>
- <li data-i="3"><b>单轮超时</b>:单轮执行设整体超时保护,避免卡死影响后续轮次。</li>
- <li data-i="4"><b>时间窗口</b>:邮件仅检索<strong>近 3 天</strong>收到的通知,天然屏蔽历史陈旧通知,避免重复告警。</li>
- </ul>
- </div>
- </div>
- </section>
- <!-- ============ 5. 模型探测 ============ -->
- <section>
- <h2><span class="no n5">5</span>模型探测机制</h2>
- <p class="lead">基于阿里云官方 dashscope SDK 发起推理请求,按 DashScope 协议判定健康状态。</p>
- <div class="cards">
- <div class="card"><h4><i>💬</i>chat · 文本对话</h4><p>文本对话类模型(如 qwen-turbo / qwen-plus / qwen-max),发送一句固定问候语验证。</p></div>
- <div class="card"><h4><i>🖼️</i>image · 图像输入</h4><p>多模态图像输入模型(如 qwen-vl 系列),附带一张示例图与固定提问验证。</p></div>
- <div class="card"><h4><i>🎙️</i>audio · 音频输入</h4><p>音频输入模型(如 qwen-audio 系列),附带一段示例音频与固定提问验证。</p></div>
- </div>
- <div class="grid2" style="margin-top:14px">
- <div class="card"><h4><i>✅</i>健康判定规则</h4><p><b>正常</b>:HTTP 200 且响应包含配置的必填字段。<br>
- <b>警告</b>:HTTP 200 但缺少必填字段(响应格式异常)。<br>
- <b>紧急</b>:HTTP 非 200(模型下线、鉴权失败、参数变更等)。</p></div>
- <div class="card"><h4><i>📋</i>可校验内容</h4><p>每个模型可配置期望响应中<strong>必须包含的字段</strong>(点路径,如输出正文、请求 ID),缺字段即视为异常,防止"返回 200 但内容不对"的假成功。</p></div>
- </div>
- <div class="note">判定口径:默认响应成功返回 <strong>HTTP 200</strong> 状态码;一旦非 200 或字段缺失,即触发本轮告警。</div>
- </section>
- <!-- ============ 6. 邮件告警与去重 ============ -->
- <section>
- <h2><span class="no n6">6</span>邮件告警与云端去重</h2>
- <p class="lead">模型下线通知"仅告警一次",通过<strong>云端发件箱留存 + 标题内嵌 Message-ID</strong> 实现无状态去重,不依赖任何本地文件。</p>
- <div class="svgbox">
- <svg viewBox="0 0 1000 560" xmlns="http://www.w3.org/2000/svg" font-family="'Microsoft YaHei','PingFang SC',sans-serif">
- <defs>
- <marker id="arr3" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#5b6b96"/>
- </marker>
- <marker id="arrc3" markerWidth="9" markerHeight="9" refX="7" refY="4.5" orient="auto">
- <path d="M0,0 L9,4.5 L0,9 Z" fill="#2dd4bf"/>
- </marker>
- </defs>
- <!-- 生命线 -->
- <line x1="150" y1="60" x2="150" y2="520" stroke="#2a3558" stroke-width="1.4" stroke-dasharray="4 4"/>
- <line x1="430" y1="60" x2="430" y2="520" stroke="#2a3558" stroke-width="1.4" stroke-dasharray="4 4"/>
- <line x1="700" y1="60" x2="700" y2="520" stroke="#2a3558" stroke-width="1.4" stroke-dasharray="4 4"/>
- <line x1="920" y1="60" x2="920" y2="520" stroke="#2a3558" stroke-width="1.4" stroke-dasharray="4 4"/>
- <rect x="70" y="22" width="160" height="34" rx="8" fill="#16214a" stroke="#fbbf24"/>
- <text x="150" y="44" fill="#fbbf24" text-anchor="middle" font-size="13" font-weight="700">阿里云通知</text>
- <rect x="350" y="22" width="160" height="34" rx="8" fill="#16214a" stroke="#38bdf8"/>
- <text x="430" y="44" fill="#38bdf8" text-anchor="middle" font-size="13" font-weight="700">监控系统</text>
- <rect x="610" y="22" width="180" height="34" rx="8" fill="#16214a" stroke="#a78bfa"/>
- <text x="700" y="44" fill="#a78bfa" text-anchor="middle" font-size="13" font-weight="700">SMTP · 云端发件箱</text>
- <rect x="840" y="22" width="160" height="34" rx="8" fill="#16214a" stroke="#f87171"/>
- <text x="920" y="44" fill="#f87171" text-anchor="middle" font-size="13" font-weight="700">通知人</text>
- <!-- 1 收件 -->
- <line x1="160" y1="90" x2="420" y2="90" stroke="#5b6b96" stroke-width="1.6" marker-end="url(#arr3)"/>
- <text x="290" y="82" fill="#e6ecff" text-anchor="middle" font-size="12">发送模型下线通知邮件</text>
- <text x="290" y="106" fill="#93a0c4" text-anchor="middle" font-size="11">落入收件箱 INBOX</text>
- <!-- 2 判断 -->
- <rect x="300" y="128" width="260" height="54" rx="10" fill="#1a2340" stroke="#38bdf8"/>
- <text x="430" y="150" fill="#e6ecff" text-anchor="middle" font-size="12" font-weight="700">匹配关键词 / 发件人过滤</text>
- <text x="430" y="170" fill="#93a0c4" text-anchor="middle" font-size="11">查询云端发件箱 → 无此 ID → 判定为新事件</text>
- <!-- 3 群发 -->
- <line x1="440" y1="200" x2="688" y2="200" stroke="#f87171" stroke-width="1.6" marker-end="url(#arr3)"/>
- <text x="560" y="192" fill="#f87171" text-anchor="middle" font-size="12" font-weight="700">群发告警,标题嵌入 [ALERT-MSG:原始ID]</text>
- <line x1="710" y1="228" x2="910" y2="228" stroke="#f87171" stroke-width="1.6" marker-end="url(#arr3)"/>
- <text x="810" y="220" fill="#e6ecff" text-anchor="middle" font-size="12">告警邮件送达通知人</text>
- <!-- 4 留存 -->
- <rect x="610" y="256" width="180" height="52" rx="10" fill="#1a2340" stroke="#a78bfa"/>
- <text x="700" y="276" fill="#e6ecff" text-anchor="middle" font-size="12" font-weight="700">告警邮件留存发件箱 Sent</text>
- <text x="700" y="296" fill="#93a0c4" text-anchor="middle" font-size="11">标题含 Message-ID,供后续查询</text>
- <!-- 5 下轮去重 -->
- <text x="430" y="352" fill="#2dd4bf" text-anchor="middle" font-size="12" font-weight="700">下一轮触发</text>
- <line x1="440" y1="368" x2="690" y2="368" stroke="#2dd4bf" stroke-width="1.6" stroke-dasharray="6 4" marker-end="url(#arrc3)"/>
- <text x="560" y="360" fill="#93a0c4" text-anchor="middle" font-size="11">查询发件箱中已告警的 Message-ID</text>
- <line x1="690" y1="396" x2="440" y2="396" stroke="#2dd4bf" stroke-width="1.6" stroke-dasharray="6 4" marker-end="url(#arrc3)"/>
- <text x="560" y="412" fill="#2dd4bf" text-anchor="middle" font-size="11" font-weight="700">命中已存在 → 跳过,不重复告警</text>
- <text x="430" y="470" fill="#93a0c4" text-anchor="middle" font-size="12">收件箱中该邮件也不再触发(近 3 天窗口内已被标记)</text>
- </svg>
- </div>
- <div class="note"><b>去重原理</b>:告警邮件标题携带原始通知的 Message-ID 并留存于云端发件箱;下一轮直接查询发件箱即可得知"这条通知是否已告警过",<b>无需任何本地缓存</b>。</div>
- </section>
- <!-- ============ 7. 告警聚合与通知 ============ -->
- <section>
- <h2><span class="no n7">7</span>告警聚合与双通道通知</h2>
- <p class="lead">模型告警是即时性的:只要本轮探测到异常,就在本轮结束时聚合推送,不跨轮积压。</p>
- <div class="grid2">
- <div class="card"><h4><i>📦</i>聚合规则</h4><p>例:10 个模型中 3 个异常 → 本轮结束后把<strong>问题模型列表及错误详情合并为一条告警</strong>,而非逐条轰炸。邮件下线事件按事件逐条独立推送(因涉及各自去重)。</p></div>
- <div class="card"><h4><i>🤖</i>钉钉群机器人</h4><p>向公司内部群推送 Markdown 告警,支持 HMAC 加签与指定成员 @提醒,@全体可选。</p></div>
- <div class="card"><h4><i>📧</i>邮件群发</h4><p>通过 SMTP 向配置的通知人邮箱列表群发;标题内嵌 Message-ID 供云端发件箱去重闭环使用。</p></div>
- <div class="card"><h4><i>🛡️</i>系统自身异常兜底</h4><p>邮箱连接失败等系统级错误<strong>仅走钉钉</strong>,避免"告警通道自身故障导致死循环刷邮件"。</p></div>
- </div>
- <div class="note blue">双通道同时推送:钉钉负责即时提醒与 @成员,邮件负责留存归档与二次通知,互为备份。</div>
- </section>
- <footer>阿里云百炼模型下线监控系统 · 技术方案(Python / dashscope SDK)</footer>
- </div>
- </body>
- </html>
|