Competency frameworks in practice: from standards to evidence to mastery

A competency framework dies the moment it stops being connected to the thing that produces evidence. In most accredited programmes the framework is a document — a PDF a validation panel signed off, a spreadsheet with codes down the left-hand column — while the assessments that would prove those competencies live somewhere else: a question bank, a marking sheet, a folder of portfolio submissions. Nothing joins the two except a coordinator who remembers how, and coordinators change jobs. The framework is not wrong; it is unreachable. So the programme runs on marks all year, and the evidence table gets rebuilt by hand in the fortnight before a review.

That rebuild is the tell. If one person can reconstruct your evidence map from memory and a spreadsheet, the map was never load-bearing. A framework earns its place only when marking a piece of work and recording evidence are the same act.

Why frameworks die

Four failure modes, and they compound.

• It was written for approval, not for use. It existed to satisfy a validation panel. It succeeded, then went into a shared drive, because nothing in the running of the programme requires opening it.
• The codes have no home. A competency is `HC4.2` in the framework, "communicating with families" in the module handbook, and criterion 3 in the rubric the marker actually uses. Three names for one thing, joined only inside somebody's head.
• Nobody owns revision. Standards change, modules get rewritten, a placement partner changes what it will sign off. The framework stays put, because moving it is nobody's job and everybody's risk.
• The grain is wrong in both directions. Half the statements are so broad that every piece of work evidences them; the rest are really criteria for one task, and they multiply until nobody reads past page one.

The cheapest repair is to rewrite every statement as something a second person could observe. "Takes a patient history and summarises it for a colleague accurately in under three minutes" can be moderated across markers and years. "Understands professional communication" cannot be moderated by anyone, which is why it survives every review and proves nothing.

What mapping actually requires

Mapping gets described as one join — competency to course — and that version is close to worthless. A course-level map records what you meant to cover. It says nothing about what an individual learner proved. The chain has to reach the item.

1. Competency to programme or module. Where the thing is taught. Everyone does this join, and on its own it evidences intent.
2. Competency to assessment. Which sitting, submission or observation tests the claim. This is where most maps stop.
3. Competency to the question, task or rubric criterion. What actually produced the mark. Skip it, and a learner who passed the paper by answering everything except the two items covering `HC4.2` has, on paper, evidenced `HC4.2`.

Each attachment needs two properties the map usually lacks: a weight and a threshold. A competency touched by one multiple-choice item and by a supervised case study should not count them equally, and the pass mark on a knowledge item is rarely the bar for competence in practice.

Then run coverage both ways, before publishing rather than after the cohort has sat the paper. Forwards: every competency has a mapped item, and the load-bearing ones have more than one instrument behind them. Backwards: every item evidences something.

A blunt test of whether the framework exists as data: can this be answered without ringing anyone?

-- one learner, one competency: the evidence and the decision it supports
select c.code,
       e.source_type,   -- exam | portfolio | observation | external | manual
       e.source_ref,    -- which attempt, which artefact
       e.score, e.recorded_at, e.recorded_by,
       m.state,         -- in_progress | mastered | expired
       m.rule, m.decided_at, m.decided_by
from competency c
join evidence e     on e.competency_id = c.id
left join mastery m on m.competency_id = c.id
                   and m.learner_id    = e.learner_id
where e.learner_id = :learner and c.code = 'HC4.2'
order by e.recorded_at;

The columns are the argument. If the records cannot fill `source_ref`, `recorded_by` and `rule`, you have marks rather than evidence, and the difference surfaces in front of a reviewer.

What counts as evidence

Evidence is anything that lets a second competent person reach the same judgement without repeating the assessment. That rules out much of what ends up in evidence tables, and lets in more than exam results.

• A graded assessment result. Strongest and narrowest — dated, attributable, already moderated, and no use for what cannot be tested on paper or on screen.
• A portfolio artefact with a judgement attached. The artefact alone is not evidence; the artefact plus a marker's decision against stated criteria is.
• A supervisor or placement observation, against a defined scale, at a known date, by a named person whose sign-off rights you can show.
• Peer review, where the competency is genuinely about working with others. Supporting evidence rather than deciding evidence, and the rule should say so.
• An external record — a prior qualification, a licence, training done elsewhere — with its provenance and its expiry, not retyped as a bare tick.
• A manual staff entry, for work done away from any system. Legitimate, and the category a reviewer samples first.

Whatever the source, an evidence record needs five things: what it evidences, where it came from, who judged it, when, and the judgement. And it has to be append-only. A record that can be edited later cannot answer the question a review turns on — was this written when the work was marked, or when the pack was assembled?

Who decides mastery, and how

Mastery is a decision. An average can make it, but only if somebody chose the rule in advance and wrote it down. Three rules cover almost everything, and a framework of two hundred competencies usually wants all three rather than whichever one the tool made easy.

• Threshold. Mastered when the weighted average across mapped evidence reaches a stated score, with a minimum number of pieces. Right for knowledge-shaped competencies where the instrument is reliable.
• Evidence count. Mastered on n independent pieces at or above the standard, ideally from two sources or two occasions. Right for performance-shaped competencies, where once may be luck and twice under different conditions is not.
• Manual. Never awarded automatically. A named assessor decides, dates it and records a reason. Right where professional judgement is the point of the qualification.

The second question is who decided. Automatic mastery does not remove the decision; it moves it to whoever configured the rule, on the date they configured it. That holds up as long as the rule and its version are stored with the transition, so "the system awarded it" resolves to a person, a rule and a date. It stops holding up when thresholds are edited in place with no history.

The third is expiry. In regulated fields, mastery that never lapses is a claim the framework cannot support: a safeguarding competency evidenced in 2019 says nothing about today. A re-certification window, with a sweep that flips lapsed mastery to expired, keeps the register honest without anyone keeping a diary.

What an accreditation review actually wants to see

Reviewers do not ask for the framework. They already have it — you sent it with the submission. They ask four things, all about traceability rather than design.

1. Show me this competency for this learner. They pick the learner, usually from the middle of the cohort. They expect the evidence, the dates, who judged each piece, the decision and the rule behind it — while sitting with you, not next week.
2. Show me coverage. Every competency assessed somewhere, the load-bearing ones assessed by more than one instrument, none resting entirely on a single multiple-choice item.
3. Show me that marking is consistent. Moderation and second marking, how rubrics were applied across markers, what the disagreements were and how they were settled.
4. Show me what changed. Version history on the framework and its thresholds: what moved, when, who approved it. A framework edited in place with no history invites the assumption you would least like them to make.

None of this requires software; programmes have satisfied reviewers on paper for decades. What software changes is the gap between producing a mark and recording evidence. In that gap the record depends on somebody remembering to transcribe it, and transcription is where evidence maps quietly turn into fiction.

What this asks of the system you run it on

Four unglamorous requirements fall out. The framework has to exist as data, with a stable code per competency. The map has to reach question and rubric-criterion level, with weights. Evidence has to be append-only and carry its source. The mastery rule has to be explicit, versioned and recorded with each decision. A system that does the first two and not the last two gives you a good curriculum map and a bad audit trail.

That is the shape of platform we build, so read this as interested rather than neutral. In Lurno a framework is a tree by subject, grade and domain, and each competency carries a code, the evidence it requires, a mastery method and an optional re-certification window. Competencies attach to an assessment and to individual questions inside it, each attachment carrying its own weight and an optional threshold override, and a publish check warns when a competency has no mapped question. A graded attempt appends an evidence record with its source and score; staff add evidence by hand for off-platform work, and evidence is never edited afterwards. Mastery is recomputed by a threshold rule, an evidence-count rule, or a manual rule that never auto-masters, and every transition writes to a hash-chained audit log.

Learners get a mastery matrix, the evidence behind each cell, and a dossier export. Frameworks published across the platform are read-only inside a tenant, so copying one into your organisation is how you get a version you can edit — which keeps the standards you started from distinguishable from your local changes to them. How mastery becomes a credential someone can verify covers the last step; what this looks like for a college covers the cohorts, prerequisites and exams around it.

The test to run this week costs an afternoon. Take one competency, pick one learner from last year's cohort, and assemble the evidence that they met it: the pieces, the sources, the dates, the marker, the decision, the rule. If that takes an afternoon, the next review will take a term. If it takes ten minutes, the framework is alive and the document in the shared drive is just its printout.