The Data Wasn't Hidden. It Was Never Written.

The Card was three lines long, and it sounded like a reporting story:

As a trainer, I want to see statistics on which quiz questions and which lessons my students fail the most, so that I can identify where they are struggling.

So I went looking for the failures. Here is what I found in the Quiz player:

QuizAttempt.correct?(question, option_id) ->
  {:noreply, answer_correctly(socket, question, qid, option_id)}

true ->
  {:noreply,
   assign(socket, :feedback, {:wrong, option_id, QuizAttempt.wrong_message(question)})}

Look at the wrong branch. It assigns a message to the socket. That’s it. Courses.record_answer/3 — the function that persists a pick — is called from answer_correctly/4 and nowhere else. A QuizAttempt’s stored answers contain only the options the Student got right.

Word Match was the same story: check/2 returned {:errors, count}, the screen rendered the count, and nothing touched the database.

There was no failure data to report on. Not hidden, not unaggregated — unrecorded.

A quiz you can’t fail has no failure rate

That reframed the whole item, and it’s worth sitting with, because it’s a trap the word “statistics” walks you straight into.

Forgia’s Quiz is a try-until-correct loop. Pick wrong, get a nudge, pick again. You cannot advance until you’re right, and you cannot finish with a wrong answer on your record. Every Student who completes a Quiz completes it at 100%.

So “the questions they fail most” cannot mean a pass rate. There is no pass/fail to count. It can only mean how many wrong picks they made before getting it right — and that’s a better measure anyway. A pass rate tells you how many people cleared the bar. The number of fumbles tells you how hard the bar was.

Which means the screen — the thing that looked like the feature — was the easy half. The real work was capturing an event that did not exist.

Passive and Active, and why the distinction earns its keep

Before writing any of it, the domain model needed a word it didn’t have. Mário’s steer, and the best idea of the session:

“Maybe we should also add an entry to context establishing a difference between Passive and Active lessons. Word Match isn’t the last type of active lesson we are going to create.”

He’s right, and the backlog proves it — True/False Questions, Sentence & Answer, and In-Video Quiz Gates are all queued, and all failable. So:

• A Passive Lesson (Text, Video) is consumed. Its Completion is self-declared: you click “Mark as complete”. Nothing in it can be gotten wrong, so it can never appear in Analytics — a dash, never a zero.
• An Active Lesson (Quiz, Word Match, …) is answered. Its Completion is earned: the server judges, and the Lesson completes only once the work is right.
• A Challenge is the atom of an Active Lesson — the smallest thing a Student can get right or wrong. A Quiz’s Challenges are its Questions. A Word Match’s are its Blanks.

That last one is the load-bearing abstraction. It means there is one event shape, not two:

flowchart TD
    subgraph Active["Active Lessons"]
        Q["Quiz
Challenges = Questions"]
        W["Word Match
Challenges = Blanks"]
        N["...the next type
declares its own Challenges"]
    end
    subgraph Passive["Passive Lessons"]
        T["Text"]
        V["Video"]
    end
    Q -->|"1 pick → 1 Attempt"| X["Analytics.Extractor"]
    W -->|"1 Check → N Attempts,
one per Blank"| X
    N -.->|"a new clause,
no new schema"| X
    T -.->|"nothing, ever"| Z((" "))
    V -.->|"nothing, ever"| Z
    X --> CA[("challenge_attempts
append-only")]
    CA --> AG["one grouped query
type-agnostic"]
    AG --> S["the Analytics screen"]

The extractor pattern-matches on the Lesson type only to pull the events out. The table, the aggregation and the screen are written against Challenge and never against lesson_type. A future Active Lesson type plugs in with one new clause and no schema change.

The rejected alternative was a coarse per-Lesson event carrying an error count. It tells you the exercise is hard. It never tells you which Blank trips people — and it reopens the whole question every time a new Lesson type ships.

Recording the correct answers too

Here’s the design decision that looks like a bug in a code review:

def handle_event("pick", %{"question-id" => qid, "option-id" => option_id}, socket) do
  # ...
  :ok = record_attempt(socket, {:pick, question, option_id})

  if QuizAttempt.correct?(question, option_id) do
    {:noreply, answer_correctly(socket, question, qid, option_id)}
  else
    {:noreply, assign(socket, :feedback, {:wrong, option_id, ...})}
  end
end

The recording happens before the branch. Every pick is written — the right ones too.

Why log a success in a failure table? Because the log then carries its own denominator. A wrong rate is wrong-picks over all picks, and if the table only holds the wrong ones, computing “all” means joining out to Enrollments and Questions and guessing how many people even saw the thing. Log everything, and the answer is count(*).

The Student sees no difference. The try-until-correct loop, the nudges, the Completion — all byte-for-byte what they were.

Word Match knew which Blanks were wrong. It just wasn’t telling.

Word Match posed a sharper problem. Its check/2 returned this:

def check(document, placements) do
  wrong = Enum.count(blanks(document), fn blank -> ... end)
  if wrong == 0, do: :all_correct, else: {:errors, wrong}
end

A count. By design — ADR 0014 says the answer key never reaches the browser, and “you got 2 wrong” is all the Student ever learns. Never which two. Otherwise you can brute-force the passage by elimination.

But Analytics needs exactly what the Student may not have. So I inverted the function rather than duplicating the comparison:

def check(document, placements) do
  wrong = Enum.count(verdicts(document, placements), &(not &1.correct))
  if wrong == 0, do: :all_correct, else: {:errors, wrong}
end

def verdicts(document, placements) do
  chips = Map.new(bank(document), &{&1["id"], &1})

  Enum.map(blanks(document), fn blank ->
    placed = Map.get(chips, placements[blank["id"]])
    response = if placed, do: placed["text"], else: ""

    %{blank_id: blank["id"], response: response, correct: response == blank["answer"]}
  end)
end

verdicts/2 knows everything. check/2 is now a fold over it that throws the detail away. One definition of “wrong”, used by both the Student-facing count and the Analytics record — they cannot drift apart, because there’s only one of them.

The privacy property is kept by call graph, not by hope: verdicts/2 has exactly two callers — check/2, which reduces it to a number, and the extractor, which writes it to the database. It appears in no template. A grep proves it.

The rule I nearly shipped without

Now the part that turned a nice-to-have into a correctness invariant.

The obvious question about an append-only log is what happens when the Instructor rewrites a Question. The obvious answer: count only the Attempts made since the edit, so the rewrite is judged on its own merits. Otherwise the old failures outnumber the new successes for months and the report actively hides the only thing the Instructor was trying to learn.

Nice. Reasonable. A quality-of-life feature. And then I looked at how Word Match mints its Blank ids:

defp blanks_from(tokens) do
  tokens
  |> Enum.filter(&match?({:blank, _, _, _}, &1))
  |> Enum.with_index(1)
  |> Enum.map(fn {{:blank, _line, category, word}, i} ->
    %{"id" => "b#{i}", "category" => category, "answer" => word}
  end)
end

b1, b2, b3 — by order of appearance. They’re positional.

Insert a Blank into an early paragraph and every later id shifts by one. b4 is “Retrospective” today and “Sprint Review” tomorrow. Count the whole log and you will confidently attribute one Blank’s failures to a completely different Blank — and the number will look perfectly plausible. There is no crash, no error, no red test. Just a wrong answer with a straight face.

The last-edit window fixes it for free, and this is why it’s ADR 0017 rather than a footnote. Any edit reparses the whole document and bumps lessons.updated_at, so every row that survives the window is guaranteed to speak the current document’s ids. The rule I’d added for the Instructor’s convenience turned out to be the thing standing between the feature and silently fabricated data.

Here it is, and it’s the only place in the codebase that expresses it:

defp windowed(lesson_ids) do
  from(a in ChallengeAttempt,
    join: l in Lesson,
    on: l.id == a.lesson_id,
    left_join: q in Question,
    on: q.lesson_id == a.lesson_id and fragment("?::text", q.id) == a.challenge_id,
    where: a.lesson_id in ^lesson_ids,
    where:
      (l.lesson_type == :word_match and a.inserted_at >= l.updated_at) or
        (not is_nil(q.id) and a.inserted_at >= q.updated_at)
  )
end

That left_join pulls double duty. A Question windows at its own updated_at — but a Question that’s been deleted finds no match at all, so its rows drop out of the aggregate while standing untouched in the log. Deleted content stops being reported without anything being erased.

And it’s visible on screen, because a window the Instructor can’t see is a window they’ll mistake for a lifetime total:

A dash is not a zero

Look again at that screenshot, at the two rows above the rewritten Question.

“transparency” shows 0%. The rewritten Question shows —.

Those mean opposite things, and conflating them is the classic analytics lie. 0% says nobody gets this wrong. — says nobody has tried it. Render an unattempted Challenge as 0% and you’re asserting something your data cannot support — and worse, in a list sorted hardest-first, a wrong rate of zero would float it to the top, right where the Instructor looks first.

So the arithmetic returns nil, never 0, for an empty sample:

def wrong_rate(wrong, total) when is_integer(wrong) and is_integer(total) do
  if total > 0, do: wrong / total
end

…and the never-attempted Challenges sort to the bottom, because they aren’t easy. They’re unknown.

That guard clause — is_integer — is a scar. Postgres hands sum() back as a Decimal, and Decimal / Decimal is an ArithmeticError. But a Decimal struct is greater than the integer 0 under Erlang’s term ordering, so a naive when total > 0 waves it straight through and blows up one line later. The coercion now happens at the database boundary, and the guard makes sure it can’t sneak past again.

Two levels of grouping, and why

The last subtlety is in the aggregation, and it comes from a correction Mário made during refinement:

“Note that percentage should count all Attempts, so even with just 2 students we could have 10% passing rate. It’d be nice to have some other metrics as well: average number of attempts and median number of attempts.”

Percentages are weighted by Attempt. Averages are per Enrollment. Those are different denominators, in the same row of the same table, and both are wanted at once.

Suppose one Student nails a Question first try and another fumbles nine times before getting it. Weighted by Attempt: 9 wrong of 11 picks — 82%, a hard Question. Averaged over the two Students’ personal rates (0% and 90%): 45%. The lucky one cancels out the struggling one, which is precisely backwards from the signal you want.

So the query counts Attempts per Enrollment first, then reduces over Enrollments — summing back up for the rate, averaging across for the rest:

per_enrollment =
  windowed(lesson_ids)
  |> group_by([a], [a.lesson_id, a.challenge_id, a.enrollment_id])
  |> select([a], %{
    lesson_id: a.lesson_id,
    challenge_id: a.challenge_id,
    enrollment_id: a.enrollment_id,
    attempts: type(count(a.id), :integer),
    wrong: type(filter(count(a.id), a.correct == false), :integer)
  })

from(p in subquery(per_enrollment),
  group_by: [p.lesson_id, p.challenge_id],
  select: %{
    total: sum(p.attempts),
    wrong: sum(p.wrong),
    enrollments: count(p.enrollment_id),
    average: avg(p.attempts),
    median: fragment("percentile_cont(0.5) WITHIN GROUP (ORDER BY ?)", p.attempts)
  }
)

The median is there because it’s the outlier guard — one Student who fumbled forty times drags the average to 11 while the median sits calmly at 2, telling you most people take two goes. And percentile_cont interpolates: the median of [1, 2, 3, 4] is 2.5, not 2. That’s a real decision, not a default, so it’s mirrored in a pure Elixir function that says so out loud — if the two ever disagree, a test fails rather than a number quietly lying.

Keyed by Enrollment, never by Student

One more thing, and it’s the reason the whole feature is safe to keep forever.

A Challenge Attempt hangs off the Enrollment, exactly as a Completion does. Never off the Student.

Three things fall out of that for free. The Course filter is an Enrollment join. The grouping grain for the average and median is one Enrollment — which is one Student in one Course, precisely the unit we want. And the GDPR posture inherits the established pattern: a scrub wipes the Profile and anonymizes the email while the Enrollment, the Completion and the Certificate stand. A Challenge Attempt now stands with them — the aggregate survives intact, and the row becomes unattributable.

The screen names no Student, anywhere. That was Mário’s call, raised unprompted:

“It’s mostly an overall feedback for me.”

And it’s the right call for a reason beyond privacy: knowing who is struggling is a different job (intervene in a person’s learning) from knowing what is failing (fix the material). This screen does the second one. The first is its own Card.

What the Instructor actually sees

Top row: the Word Match Blank for “commitment” — 100% wrong, and the word that stole it was “promise”. That’s not a difficulty score, that’s a diagnosis. The distractor is too close to the answer, and now the Instructor knows exactly which word to change.

Below it, “When does a Sprint end?” at 67%, with “When the Product Owner says so” pulling half the wrong picks.

Every row carries its sample size — 2 wrong of 3 attempts, across 1 student — because a thin number presented as a fat one is how confident, wrong decisions get made. There’s no minimum-sample threshold and no suppression. The median is the outlier guard, and the sample size is right there on the screen. The Instructor can see the number is thin. They don’t need to be protected from it.

The story took a day. About four hours of it went into a screen, and the rest into the realization that the interesting question — where are my students struggling? — had never had an answer, because nobody had ever written it down.

Two Cards came out of it, both deliberately left on the table: “See which Students are struggling” (the named drill-down, a teaching intervention) and “Lesson drop-off funnel” (where they quit — which needs a Lesson-view event that, right now, also doesn’t exist).

I have a suspicion about how that second one is going to go.