Classifying a job description into one of the NOC 2021 unit groups is the one place in MaplePath where a wrong answer is plausible and consequential - it decides TEER, and TEER decides program eligibility. So retrieval is measured, not asserted. 300 job descriptions, hand-labelled with their correct NOC code, run through four retrieval methods with identical queries.
Higher NDCG@10 means the correct code sits nearer the top of the list. The shipped method is the second row.
One row per labelled description, from examples_report. NDCG and hit are for the shipped vector method.
| Job title | Case type | NDCG | Outcome |
|---|---|---|---|
| Software Developer | exact_title | 0.966 | Top 10 |
| Software Engineer | close_sibling | 0.861 | Top 10 |
| IT Support Technician | vague_title | 0.788 | Top 10 |
| Bookkeeper | teer_distinction | 0.411 | Top 10 |
| Registered Nurse | exact_title | 0.724 | Top 10 |
| Licensed Practical Nurse | close_sibling | 0.372 | Top 10 |
| Cook | close_sibling | 0.478 | Top 10 |
| Electrician | exact_title | 0.723 | Top 10 |
| Transport Truck Driver | exact_title | 0.642 | Top 10 |
| City Councillor | vague_title | 0.000 | Missed |
| Deputy Minister | exact_title | 0.000 | Missed |
| President, Trucking Company | vague_title | 0.000 | Missed |
| Hospital CEO | vague_title | 0.000 | Missed |
| Finance Manager | exact_title | 0.486 | Top 10 |
| HR Manager | exact_title | 0.429 | Top 10 |
| Procurement Manager | close_sibling | 0.384 | Top 10 |
| Bank Branch Manager | exact_title | 0.567 | Top 10 |
| Marketing Manager | close_sibling | 0.586 | Top 10 |
| Marketing Specialist | close_sibling | 0.267 | Top 10 |
| Accountant | teer_distinction | 0.652 | Top 10 |
Labels live in data/noc_eval_labels.csv. This report is produced live by the /evaluate endpoint on each visit.