The Impact of AI on
Software Engineering

The two big impacts of AI on software

Deterministic vs. probabilistic

AI engineering is mostly about constraining, grounding, and evaluating that output.

Building on AI models: Core concepts

LLMs: same prompt, different output

client = openai.OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama")

response = client.chat.completions.create(
    model="gpt-oss:20b",
    messages=[{"role": "user",
        "content": "Write a haiku about recursion"}])
print(response.choices[0].message.content)
					

🔗 llm.py

Context engineering

Sending context to the LLM to personalize the response and ground responses in domain-specific data.

RAG: Retrieval-Augmented Generation

RAG example: a syllabus Q&A bot

def answer(question):
    sources = "\n\n".join(search(question))
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {"role": "system", "content": "Answer ONLY using these syllabus sections. "
                f"Cite the section title in square brackets.\n\n{sources}"},
            {"role": "user", "content": question}])
    return response.choices[0].message.content
					

🔗 rag.py · syllabus.md

Evaluations: how do you test this?

def test_penalty():
    assert penalty(0) == 0
    assert penalty(1) == 10
    assert penalty(2) == 20
							

question = "When is the final project due?"
answer_text, sources = answer(question)
verdict = client.chat.completions.create(
    model=MODEL,
    messages=[{"role": "user", "content":
        "Is every claim in this answer supported "
        "by the sources? Start with PASS or FAIL, "
        "then give a one-sentence reason.\n"
        f"Sources: {sources}\nAnswer: {answer_text}"}]
    ).choices[0].message.content
							

Bulk evals: one question isn't enough

Always run evals on a diverse dataset of inputs:

Structured outputs

Instruct the LLM to output data that conforms to a typed schema:

class CodeFeedback(BaseModel):
    concepts_used: list[str]
    bugs: list[str]
    hints: list[str]

completion = client.chat.completions.parse(
    model=MODEL,
    messages=[{"role": "user", "content":
        f"Review this student code:\n{code}"}],
    response_format=CodeFeedback)
feedback = completion.choices[0].message.parsed
							

def average(scores):
    total = 0
    for i in range(1, len(scores)):
        total += scores[i]
    return total / len(scores)
							

Tool calling

Give the LLM access to tools, like local functions or remote MCP servers:

The model never runs code. It only suggests a tool call for your code to make.

Tool calling: the code

tools = [{"type": "function", "function": {
    "name": "get_weather_alerts",
    "description": "Get active weather alerts from NOAA for a US state",
    "parameters": {"type": "object",
        "properties": {"state": {
            "type": "string", "description": "Two-letter state code"}},
        "required": ["state"]}}}]
					

response = client.chat.completions.create(
    model=MODEL, tools=tools,
    messages=[{"role": "user",
        "content": "Any weather alerts in California?"}])
tool_call = response.choices[0].message.tool_calls[0]
args = json.loads(tool_call.function.arguments)
alerts = get_weather_alerts(**args)
						

🔗 tool_calling.py

Agents: tool calling in a loop

The agent loop in code

while True:
    response = client.chat.completions.create(
        model=MODEL, messages=messages, tools=tools)
    message = response.choices[0].message
    messages.append(message)
    if not message.tool_calls:
        break
    for tool_call in message.tool_calls:
        args = json.loads(tool_call.function.arguments)
        result = tool_functions[tool_call.function.name](**args)
        messages.append({"role": "tool",
            "tool_call_id": tool_call.id, "content": result})
print(message.content)
							

Other kinds of models

Embedding models: text in, numbers out

response = client.embeddings.create(
    model="nomic-embed-text",
    input=["dog", "puppy", "cat", "perro", "spreadsheet"])
vectors = [item.embedding for item in response.data]
					

Similar meanings get similar vectors, measured with cosine similarity. Not probabilistic: same input, same vector. But very dependent on training data…

🔗 embeddings.py

Training data shapes what's “similar”

Most similar words to “dog”, out of 1000 common words:

Try it: pamelafox.github.io/vectors-comparison

Source: A visual introduction to vector embeddings

Multilingual support varies a lot by model

With the English-only model, “perro” is about as similar to “dog” as “spreadsheet” is.

AI coding agents

A coding agent is an agent with tools specific to code generation:

A real agent session: building this deck

Coding agents keep getting better, faster

Length of software tasks (in human time) the best AI model completes 50% of the time:

Source: METR time horizons v1.1, updated May 2026. Doubling about every 4 months since 2023.
More benchmarks: Artificial Analysis Coding Agent Index · SWE-bench

Developers are actually using agents

Agent-created PRs: 17.8M in March 2026 alone, quadrupled in six months. By August, GitHub had more agent-authored PRs than human-authored ones.

Sources: GitHub Blog, “The August 17 outage, and the work ahead”; The Pragmatic Engineer, “The state of the tech industry in 2026”

Agents finish big projects much faster

Source: The Pragmatic Engineer, “The state of the tech industry in 2026”

How software engineers work now

“I have 5 terminal tabs… I also run 5-10 Claudes on Claude Web, in parallel with my local Claudes.”
— Boris Cherny, creator of Claude Code

See also: Nicholas C. Zakas, “Five software engineering roles for working with AI”

From IDE to agent dashboard

Screenshots from Parallelize your development with GitHub Copilot

Agents excel at verifiable tasks

Agents are weaker at judgment calls

Verification is the new bottleneck

Sources: Stack Overflow 2026 Developer Survey; “These Aren't the Reviews You're Looking For” (May 2026); Artificial Analysis reward hacking detection

Agents struggle with system design

Sources: SWE-bench Verified and SWE-Bench Pro leaderboards (mini-SWE-agent); Demirer, Musolff & Yang, “Writing Code vs. Shipping Code” (NBER, June 2026)

Coding agents are improving fast

Sources: study of 567 Claude Code PRs (2025); Popescu et al. (April 2026)

More software is being built than used

Code is cheap. Knowing what to build isn't.

Source: Demirer, Musolff & Yang, NBER w35275 (2026)

Build software that people want

Source: Laurie Voss, We are all Product Engineers now

The software engineering lifecycle

Writing code is only one step

Which steps still need humans?

Software careers are evolving

Each layer lowered the barrier to building

“The hottest new programming language is English” — Andrej Karpathy, January 2023

Not a new idea, just newly possible

Personal vs. mass-market software

Only the baker knows what her bakery needs. Now the baker can build it herself.

At AI-native companies, “the salespeople are shipping (at least internal tools and automations for themselves).” — Yoni Rechtman

Faster to build it than to find it

“I was trying to find this one website that lets my kids record audio and plays it back with funny effects. Then I realized, screw it, its faster to vibe it than to find it! Two minutes later, boom, my kids can squirrel themselves to their hearts desires.”

— @pamelafox, Sept 3, 2026

Personal software can get ambitious

A ChatGPT-style app my friend Matt and I built, so we can pay for just the tokens we use instead of a subscription.

• Web search
• Image and voice input
• Conversation history
• Token counts on every chat

mattgotteiner.github.io/responses-chat

What still takes software knowledge

Anyone can describe what they want, but building something that works well still takes software engineering skills:

Skills that matter more than ever

Whether students become software engineers or build software for themselves:

Problem design

Figuring out what to build, and describing it precisely enough that someone (or an agent) can build it.

CSTA: 1B-AP-13, 2-AP-15, 1A-AP-12 · AP CSP Create task: describe the program's purpose

Computational thinking

Decomposition, abstraction, pattern recognition, and algorithms: still the foundation, because they're how you steer agents.

Open question: how much should students still write by hand to build a mental model? We still teach arithmetic, even with calculators.

Systems design

How the pieces fit together: architecture, data models, interfaces, tradeoffs.

CSTA: 3A-AP-23 (document design decisions)

Verification

Checking that it actually works: tests, evals, security, accessibility, and whether it's what users meant.

CSTA: 2-AP-17, 3A-AP-19, 3A-AP-21, 3A-IC-25 · Practice P6: Testing and Refining Computational Artifacts

Judgment and collaboration

These matter even more when anyone can build anything quickly.

Inspired by Yoni Rechtman, There will only be four jobs

You already teach this

The shift is in emphasis, not a new subject: from writing the code to the steps around it.

CSTA K-12 Computer Science Standards

CS for all students

So they can build the software they need,
and know enough to build it well.

The CSforALL movement matters more than ever.

Thank you!

Slides:
pamelafox.github.io/my-py-talks/impact-of-ai-on-cs

Demo code:
github.com/pamelafox/my-py-talks/tree/main/impact-of-ai-on-cs

Find me online: