Nanny Agency JobsAn AI-generated shortlist should be a review aid whose reasoning can be checked against the role and candidate records. It should not become an unexplained ranking that staff accept because the first few names look plausible. Before using it, define the permitted task, the evidence it may read and the person responsible for the decision.
This guide proposes an internal workflow for reviewing a defined candidate set against a confirmed brief. It does not claim that any particular nanny agency uses automated shortlisting, recommend a hiring algorithm or establish legal compliance for an individual system.
Define the decision before running the tool
Write down what the agency is trying to decide. “Identify which of these reviewed candidates need an availability clarification” is a narrower task than “Find the best nanny.” Specify the role, the candidate set and the next permitted action. An internal comparison need not authorize rejection messages or family introductions.
NIST's AI Risk Management Framework calls for documented responsibilities and clearly differentiated human-AI oversight roles. NIST's AI RMF Core. Apply that principle by naming the recruiter who checks the evidence, the person who approves the shortlist and the route for resolving disputed information.
Agree the limits before viewing the output. If the system cannot show the underlying evidence or staff cannot inspect relevant records appropriately, hold the decision. A polished explanation written after a ranking is not necessarily a reliable account of how that ranking was produced.
Translate the brief into reviewable criteria
Use criteria confirmed for the actual opening. Separate required conditions from preferences and unresolved questions. “Available for the stated weekday schedule” is a checkable condition. “Seems like the right energy” is not a useful instruction to an automated comparison tool.
For each criterion, identify what evidence would support it and what would require follow-up. A candidate's work history might support relevant experience, but it does not establish present availability. A course entry does not automatically establish current certification. Do not let the model infer missing facts from age, names, photographs or unrelated personal details.
Review the criteria with the appropriate people before use, including relevant legal and fairness considerations. Human involvement alone does not make an unsuitable criterion acceptable. Keep the tool away from unsupported personality judgments or proxies for characteristics that are irrelevant to the role.
Build an evidence matrix instead of a mystery score
Use this original NAJ matrix for one role and a limited candidate set. Restrict access to the people authorized to review those records.
- Criterion: A confirmed requirement or clearly labeled preference.
- Evidence reference: The relevant record, section and date, with only the necessary extract.
- Evidence state: Supported, conflicting, not evidenced or awaiting an update.
- Candidate clarification: The specific factual question that remains.
- Reviewer assessment: The human conclusion and its stated basis.
- Next action: Continue review, request clarification or another approved step, with an owner.
Avoid adding a numerical score simply to make the table sortable. If the agency uses a scoring method, it needs a defensible, reviewed basis and known limitations. A model's invented weights are not a substitute for that work.
Keep machine suggestions visually separate from reviewed decisions. A cell should not change from “not evidenced” to “does not meet requirement” without supporting information and a responsible review. Missing records can reflect an incomplete file rather than a candidate's actual experience.
Work through a fictional missing-evidence case
Fictional worked example: An opening requires availability on specified weekdays and relevant school-pickup experience. Candidate A's current statement confirms the schedule but the résumé does not mention pickups. Candidate B's résumé describes pickups, while the available schedule statement is from an earlier search.
An AI summary places B first and describes A as lacking the required experience. The recruiter checks the matrix. A's experience is not evidenced in the selected document; it has not been established as absent. B's availability is not current enough to treat as confirmed.
The recruiter creates two focused clarification tasks. A is asked about relevant pickup responsibilities, and B is asked to confirm current availability for the stated schedule. Neither question reveals the other candidate's records. The shortlist remains pending rather than converting incomplete evidence into a final ranking.
After the replies arrive, the recruiter records the new sources and makes the actual assessment. The value of the matrix is that another authorized reviewer can see what changed and why, without relying on the model's original ordering.
Review both inclusions and omissions
Checking only the proposed finalists leaves a gap. Review who was left out and whether the same criteria were applied consistently. Look for candidates excluded because information was missing, wording differed or a relevant record was not included in the tool's input.
Test the workflow with fictional or appropriately authorized examples before relying on it. Include a candidate with equivalent experience described differently, a stale availability statement and a conflicting record. Check whether the system preserves uncertainty and points to the correct source. These are useful tests of this workflow, not a certification that the system is unbiased.
Record corrections and recurring failure patterns. If staff routinely have to reconstruct the evidence from scratch, narrow the automated task. It may be useful for locating passages while being unsuitable for recommending an order.
Control access and own the final action
Limit the tool's access to the approved records needed for this role. Keep sensitive screening material and unrelated candidate information out of the comparison unless a separately authorized process requires it. Review who can view the output, export it or send it onward.
Require a human decision before a shortlist becomes a family-facing introduction or a candidate communication. Record the criteria, evidence version, unresolved questions and reviewer. Do not describe a candidate as rejected, contacted or submitted merely because a tool proposed that action.
When a candidate corrects a material fact, identify which active comparisons used it and update them through the agency's process. Preserve enough history to explain the correction without keeping unnecessary duplicate files indefinitely. A useful shortlist is one the agency can account for: what evidence it considered, what remained uncertain and who made the decision.
Sources
NIST: AI Risk Management Framework Core. The source supports the narrow responsibility and oversight statement. The matrix, fictional example and review workflow are original NAJ editorial guidance, not a validated selection method or legal compliance determination.