Po-Shen Loh: Why Operators Must Learn to Grade AI Output

TL;DR

The operator’s job has fundamentally changed. We no longer get paid to execute repetitive tasks, because artificial intelligence does that instantly. Our real value now lies in evaluating machine output and knowing whether it is actually correct. This requires treating these models as research tools and doing the unglamorous field work to test them in reality.

The Shift from Executor to Evaluator

For decades, the system trained us to do the homework. Today, an operator must learn to grade the homework.

When a problem arises, the immediate reflex is to ask an AI model. The human role is no longer to generate the raw output. If a task is repetitive, a machine should run it. I am still figuring out exactly how much trust to place in these models. But I know this: operating the machine requires analytical thinking. You must understand whether the output is correct.

Consider how a good software engineer approaches an unfamiliar task. They do not expect a forum like Stack Overflow to provide the exact code they need. They use it to research methods. We must treat AI the same way. It suggests an approach. The operator evaluates the suggestion, spots the errors, and makes the final decision. The value you bring to your business is your judgment, not your typing speed.

Audit your daily operations this week and list three recurring tasks you currently execute that you should only be grading.

Unglamorous Field Work Drives Discovery

You cannot figure out what people need by sitting in an office theorising about it. You have to step into the real world.

When Po-Shen Loh wanted to understand education gaps, he did not run digital surveys. He bought audio equipment, boarded an overnight bus, and gave free math talks in public parks. He stood in front of crowds of 50 to 100 people, city after city. This forced direct interaction with thousands of parents. He found the actual pain points because he did the unscalable, unglamorous work himself.

Later, when he wanted to help a network of disadvantaged schools, he did not just write a cheque. He asked to go in and teach sixth grade. Real discovery happens when you experience the friction directly. You build better models when you understand exactly how the people you serve actually behave.

Stop guessing what your clients want and spend two hours this week doing the manual work yourself to see exactly where the process breaks.

Questions to Consider

1. What is one recurring task in your business where you still act as the executor instead of the evaluator?
2. How do you currently test the accuracy of the artificial intelligence outputs in your daily workflow?
3. When was the last time you stepped out of the owner role to perform the ground-level work yourself?
4. Are you building your new offerings based on theory, or on friction you personally experienced in the field?

Quotes

• “People used to go to school to learn how to do the homework. Today everyone needs to learn how to grade the homework.”
• “You cannot create value if you don’t interact with people. You cannot just theoretically think about the value.”
• “In the real world, when you’re doing a task, if it’s repetitive, you should get ChatGPT to do it.”