Nanny Agency JobsUse fictional records to find missed evidence and inconsistent outputs, then assign human review of any real recruiting decision.
Define the bounded test
An agency can test whether a search or drafting tool handles comparable fictional records consistently. The test is a way to find faults, not a certificate of fairness or legal compliance. Do not use real candidate or household files for an experimental bias check.
Pick one approved administrative task, such as retrieving an experience statement with its source. Write the expected evidence and unacceptable behavior before running the tool. Keep hiring criteria and decisions with accountable people.
Build synthetic pairs around evidence
Create invented profiles with the same relevant experience expressed in different words. Keep the expected result and reason for each pair in the test card. Include a missing document, an outdated availability note and a contradictory recruiter note. Examine whether the tool retrieves the same relevant passage, flags uncertainty and cites the correct source.
Do not use protected traits as selection preferences or let a generated score decide who is introduced to a family. EEOC small-business guidance states that recruiting and hiring decisions must not rest on protected characteristics. Any real legal or statistical assessment needs qualified review in the applicable jurisdiction.
Record failures and an owner
Use a test card with scenario ID, expected evidence, output, source citation, failure type, reviewer and corrective action. A missed record and an invented fact are distinct failures. Repeat the same scenario after a prompt, vendor or index change.
Fictional example: two invented résumés describe the same school-age care duties in different language. The search returns only one. A reviewer records a retrieval miss and checks the underlying index; the omitted fictional record is not treated as evidence that a person lacks experience.
Set the limit before real use
If the tool cannot support source checks, pause that use case. Even if it passes this small test, restrict it to the tested task, permit human challenge and monitor errors. NIST’s AI RMF Core describes testing and oversight as continuing governance work, not a one-time approval.
Keep the test results separate from any real applicant’s file. Resolve employment policy, accessibility and applicable legal questions with competent staff and advisers before any consequential deployment.