Threat Signal Verification and Risk Indexing | Hust Media

Threat Signal Verification and Risk Indexing

This note shows how threat signals, external feeds, AI verification, and post-audit indexing improve evidence quality, supporting platform growth through cleaner, searchable risk records.

Written date: 06/01/2025 17:18:41Engineering Notes

Introduction

Threat Signal Verification & Indexing belongs to the Validation & Monitoring Layer of a web platform. It sits after signal intake and before lookup or monitoring. You usually meet this topic when a website receives many reports or external data sources for filtering, review, and lookup.

- Technical context: This workflow includes signal ingestion, evidence verification, post-audit review, and risk indexing.

- Technical benefit: It improves evidence quality, reduces repeated review work, and makes risk records easier to search, audit, and monitor.

Practical Notes

In 2021, I faced a weak XAMPP exposure: the website database was deleted, and the attacker demanded 100 USD to return access. After recovery, I reinstalled Windows Server 2019, renewed passwords across data storage points, mapped key data locations, and used GoCron to export the database every two days. After that, the system stayed stable. Readers may meet this topic when a website gets suspicious reports or repeated signals that need review before lookup. The notes below explain threat verification and risk indexing.

Data Ingestion (Community Signals)

Community Signals are ingested from user reports and external feeds, then converted into structured cases for verification and lookup across phone numbers, bank accounts, websites, social profiles, emails, URLs, and evidence. The diagram below shows the main verification and indexing flow for those signals.

All examples in this note use synthetic or masked indicator references. They are included only to explain the validation workflow and do not represent real personal, financial, or account data.

Threat Signal Process

How threat signals move through verification and indexing

Indicators: sample_phone_ref, sample_bank_ref, sample_social_ref, sample_domain_ref, sample_email_ref, URLs. Evidence: screenshots, chat logs, payment proof, media files, report descriptions. Normalization: Vietnamese phone numbers use canonical +84, aliases are mapped, and duplicated identifiers are checked. Storage: new cases enter SQL with Processing status before search exposure.

PHP

$phone_ref = normalize_indicator_ref($phone_ref);
INSERT INTO scam_check SET indicator_ref='$phone_ref', status='Processing';

External feeds use a crawler pipeline. Python loads report lists, downloads detail pages, stores raw HTML, then parses identifiers, phone numbers, accounts, bank names, descriptions, images, and update time into JSON. PHP reads the JSON, checks duplicates, creates SQL case records, maps image paths, and queues the case for AI verification.

FLOW

Community Signals / External Feeds
-> Normalize phone, bank, social, website, email
-> Check duplicate identifier
-> Save SQL case + evidence
-> Queue for AI verification

The full flow connects user reports, crawler feeds, normalization, SQL case storage, queue handling, and AI verification.

AI Verification (Semantic + OCR Evidence)

AI verification filters noisy, duplicated, or unsafe reports before public exposure. Reports pass input validation, rule checks, spam limits, semantic review, OCR/evidence checks, and queue-based approval.

Input quality layer:

PHP receives JSON and validates required fields. Rule engine blocks malformed or repeated reports. Daily limits reduce spam. Base64 evidence images are saved as physical files and linked to the case. AI checks names, bank references, descriptions, and evidence text.

Valid reports enter orders with category_code = scams_check and Processing status. A batch worker locks records as In progress, gathers text/images, sends them to AI, then updates the case as Completed or Canceled.

JSON

{
  "signal_type": "bank_account_reference",
  "indicator_ref": "sample_bank_ref_001",
  "evidence": {
    "source_type": "ocr_image",
    "ocr_text": "SAMPLE OCR CONTENT - ACCOUNT REFERENCE MASKED",
    "semantic_score": 0.82
  },
  "review": {
    "status": "validated",
    "category": "payment_risk_signal"
  }
}

Only validated cases move from Processing to Completed, keeping unverified reports out of public results.

Real-time Post-Audit

Post-audit keeps reports clean, searchable, and reusable. Small batch workers use status gates so completed steps are not repeated and failed steps can retry.

process_scam_url: link extraction. process_json_url: JSON normalization. process_ai_url: AI-based indicator mapping. Invalid or low-quality descriptions fall back to scam_url = []. Each worker updates its own process flag.

If scam_url is empty but the description is valid, AI extracts links and stores normalized JSON in scam_description_1.scam_url.

SQL

SELECT *
FROM scan_process
WHERE indicator_hash = 'sample_indicator_hash'
LIMIT 4;

PHP

$text_vip = get_link_ai($description_scam);

Parsed indicators are mapped into lookup indexes:

Website/domain → sample-domain.test in MongoDB website index. Email → [email protected] in normalized email index. Facebook/YouTube/Telegram → sample_social_profile resolved to stable social ID. MongoDB stores id_scam_check for traceability. MySQL keeps full case details, descriptions, account data, status, and audit history.

FLOW

Signals
-> Extract links
-> Normalize JSON indicators
-> Map website / email / social
-> Index in MongoDB
-> Keep full case in MySQL

MongoDB keeps lookup fast, while SQL stores the full record for review, masking, and audit.

Threat Analysis Outcomes

Validated reports are promoted into a Threat Intelligence index for real-time lookup. A PHP rule engine classifies input before selecting the lookup path.

Website/email: normalize sample-domain.test or [email protected], then find linked id_scam_check in MongoDB. Social: resolve sample_social_profile into a stable social ID. Phone/account: normalize masked_indicator_ref and match SQL fields such as phone/account reference, bank reference, name reference, and URL. Response: return only Completed cases and mask sensitive fields before public output.

PHP

$found = find_website_entry($mongo_results, $domain);
$found = find_scam_social_entry($mongo_results, $id_social);
$account_name = substr($account_name, 0, 5) . '***';

Validated signals support real-time lookup for suspicious phone numbers, accounts, domains, emails, and social profiles. The system keeps traceable case history through id_scam_check, reduces false positives through AI review, and exposes only masked, verified data for stable platform operation.

CLOSING NOTES

Reader Value

Readers can use this pattern to organize community signals, external feeds, and OCR evidence into a cleaner validation flow for their own projects over time. In practice, it helps reduce noisy submissions, speed up review, and improve lookup reliability for stable operation in scalable web systems.

Conclusion

This threat detection design combines structured ingestion, AI-assisted verification, post-audit control, and indexed lookup into one consistent monitoring layer. It strengthens system integration and supports stable operation across connected platform components.