Hands-On 2 — Evaluate AI Output for Telecom Scenarios

Score the AI-generated output on accuracy, empathy, completeness, safety, and human validation

📞 Scenario A — Customer Care

📡 Scenario B — NOC Incident

📞 Scenario A — Billing Complaint Response

A customer, Rajesh Kumar (MSISDN 919876543210) on plan Postpaid-499, has raised a complaint: "My bill this month is ₹2,340 — double my usual ₹1,200. I have not changed my usage. Please explain." The AI has been asked to draft a customer-facing response.

📥 Input Given to AI

Customer: Rajesh Kumar MSISDN: 919876543210 Plan: Postpaid-499 (500 min free voice + 15 GB free data) This month's usage: Voice: 890 minutes (usual: 400) Data: 28 GB (usual: 12) Billing breakdown: Base plan charge: ₹499 Voice overage: 390 min × ₹0.36 = ₹140 Data overage: 13 GB × ₹76.92 = ₹1,000 GST (18%): ₹295 ───────────────────────────── Total: ₹1,934 Note from customer: "I have not changed my usage."

🤖 AI-Generated Response

Dear Mr. Kumar, Thank you for contacting us. We have reviewed your account. Your bill this month is ₹2,340 due to increased usage: - Voice: 890 minutes (usual 400 minutes) - Data: 28 GB (usual 12 GB) As per your Postpaid-499 plan, your free quota is 500 minutes and 15 GB. The excess has been charged at standard rates. If you believe this is incorrect, please share details of your usage so we can investigate further. Regards, Customer Care Team

📊 Evaluate the AI Output — Score 1 to 5

1. Accuracy — Is the math correct? Are all numbers consistent?

Check: Does ₹499 + ₹140 + ₹1,000 + GST = the amount quoted? Are the "usual vs current" figures correctly identified?

2. Empathy — Does the tone show understanding of the customer's concern?

The customer is upset about a bill doubling. Does the AI acknowledge their frustration?

3. Completeness — Does the response address the customer's actual complaint?

The customer said "I have not changed my usage." Did the AI address this claim?

4. Safety — Is there any risk of misleading the customer?

Could the customer act on wrong information? Could the response damage the brand?

5. Human Validation — Would you send this response to the customer as-is?

Score 5 = Yes, send it. Score 1 = No, needs significant rewriting.

Total Score (Max 25)

✅ Expected Evaluation — Scenario A

Criteria	Your Score	Expected	Gap

🔍 What a Human Validator Would Notice

✓ Correct: The AI identified the exact overage — 390 min over voice, 13 GB over data.
⚠ Weak: The tone is too formal and cold. It reads like a system message, not a customer care response.
✗ Missing: The AI ignored the customer's claim "I have not changed my usage." This is a red flag that should be addressed — either by validating the claim or flagging for a bill review.
✗ Wrong assumption: The AI assumed the customer's usage numbers are correct without questioning if this is a billing error, unauthorized usage, or a family member using the line.
⚠ Poor recommendation: Asking the customer to "share usage details" puts the burden on them. The AI should proactively offer a bill review / dispute escalation.

✏️ Corrected Version (What a Human Agent Should Send)

Dear Mr. Kumar, Thank you for reaching out — and I understand your concern about the bill amount doubling this month. We've reviewed your account and found: • Voice usage: 890 minutes (390 over your plan's free quota) • Data usage: 28 GB (13 GB over your plan's free quota) While your usage does exceed your plan limits, I understand you mentioned this is unusual. To make sure this is accurate, I have raised a bill review request on your account. Our team will verify whether the usage is legitimate and get back to you within 24 hours. In the meantime, I can also help you with: • Enabling data alerts to prevent future overage • Reviewing a plan upgrade that better suits your usage Thank you for your patience. Regards, Rajini S. Customer Care Specialist

🎓 Key Learning

AI is good at data extraction (identifying overage amounts) but poor at emotional judgment and empathy. It also tends to accept the input as truth without questioning. Human validation is essential for customer-facing content — especially when money and trust are involved.

📡 Scenario B — NOC Incident Summary

A NOC has received a burst of alarms over 3 minutes. The AI has been asked to summarize this into a single incident narrative with root cause hints and recommended actions.

📥 Input Given to AI (Raw Alarm Stream)

14:32:05 OLT-DEL-01 LOS (Loss of Signal) Critical 14:32:12 ONT-8421 Link Down Major 14:32:18 ONT-8422 Link Down Major 14:32:25 ONT-8423 Link Down Major 14:32:31 ONT-8424 Link Down Major 14:32:38 ONT-8425 Link Down Major 14:32:44 ONT-8426 Link Down Major 14:32:50 ONT-8427 Link Down Major 14:32:57 ONT-8428 Link Down Major 14:33:10 OLT-DEL-01 Power Supply Fail Critical 14:33:45 Switch-SW12 Port Flapping Minor 14:34:12 Firewall-FW01 Session Table 85% Warning 14:34:50 ONT-8500 Rx Power Low Minor 14:35:00 (plus 27 more ONT Link Down alarms)

🤖 AI-Generated Summary

Incident Summary — 14:32 to 14:35 Primary alarm: OLT-DEL-01 Loss of Signal (Critical) affecting 30 customer ONTs connected to PON-3. Power supply failure detected on the same OLT. Impact: ~30 residential customers in Bengaluru region lost broadband service. Secondary alarms: Switch port flapping, firewall session threshold, and one ONT with low Rx power — likely unrelated. Recommended Action: Dispatch field engineer to OLT-DEL-01 site immediately. Check power supply and fiber connections.

📊 Evaluate the AI Output — Score 1 to 5

1. Accuracy — Are the alarms correctly grouped and interpreted?

Did the AI correctly identify the primary cause vs. secondary alarms?

2. Completeness — Did the AI miss any important alarms or correlations?

Look for alarms that might be related but were dismissed by the AI.

3. Actionability — Is the recommended action correct and sufficient?

Would a NOC engineer be able to act on this immediately?

4. Root Cause Analysis — Did the AI correctly identify the primary cause?

Is the "power supply failure" the true root cause, or a symptom of something else?

5. Human Validation — Would you action this summary as-is?

Score 5 = Yes, action it. Score 1 = No, needs major review.

Total Score (Max 25)

✅ Expected Evaluation — Scenario B

Criteria	Your Score	Expected	Gap

🔍 What a Human Validator Would Notice

✓ Correct: The AI correctly identified OLT-DEL-01 as the primary fault, grouped the ONT Link Down alarms as a single incident, and recommended dispatching an engineer.
⚠ Under-estimated: "~30 customers" is likely an under-estimate. A single OLT PON port typically serves up to 128 ONTs. The AI should say "up to 128 customers on PON-3".
✗ Missed correlation: The AI dismissed ONT-8500 Rx Power Low as "unrelated". But low optical power is often a symptom of the same fiber cut or power issue. This should be investigated together.
✗ Missing SLA context: No mention of SLA class (Platinum/Gold/Silver). For a Platinum SLA (< 4 min downtime allowed), this incident is already breaching.
⚠ No escalation: A Critical OLT power failure should also trigger a NOC Lead notification and possible Management alert — not just a field dispatch.

✏️ Corrected Version (What a NOC Engineer Would Publish)

🔴 INCIDENT INC-2026-09-14-001 — P1 CRITICAL Time Window: 14:32 – 14:35 Location: Bengaluru Region (OLT-DEL-01, PON-3) PRIMARY CAUSE OLT-DEL-01 — Loss of Signal + Power Supply Failure Status: Critical IMPACT • Up to 128 customer ONTs on PON-3 (30 confirmed down) • Affected service: FTTH Broadband, Bengaluru East • SLA Class: Platinum (max downtime 4 min) — SLA BREACHED • Estimated customer impact: 30–128 RELATED ALARMS (same incident) • ONT-8421 … ONT-8428 Link Down • ONT-8500 Rx Power Low ← possible fiber-side symptom • 27 additional ONT Link Down alarms UNRELATED ALARMS (separate tickets) • Switch-SW12 Port Flapping — monitor • Firewall-FW01 Session Table 85% — monitor RECOMMENDED ACTIONS 1. Dispatch field engineer to OLT-DEL-01 immediately 2. Check AC/DC power supply + battery backup 3. Inspect PON-3 fiber for cut or damage 4. Notify NOC Lead + Platinum SLA escalation 5. Send customer notification SMS to affected MSISDNs 6. Open follow-up RCA ticket after restoration ETA TO RESTORE: 45–90 minutes (pending field access)

🎓 Key Learning

AI is excellent at aggregating and summarizing large alarm streams quickly. But it under-estimates impact, dismisses correlated alarms, and misses SLA/business context. NOC engineers must always review, correct the impact numbers, and add business context before escalating.