Then I started asking myself the boring questions. What if it writes a DELETE? What if it makes up a column? And the scary one: what if the query runs fine, returns a nice clean number, and that number is wrong?

I'm Mritunjay. I'm finishing an Associate of Science in AI at Saras AI Institute, and one of my capstone ideas is DataCopilot, a tool where you upload a dataset, ask questions in normal language, and get answers without writing SQL. This post is the first build log. It isn't the finished product. It's the skeleton, plus the one design rule I've decided to keep:

The LLM only suggests SQL. My code decides whether it runs.

Why not just let the model run queries?

Because even the best systems are still wrong a lot of the time on realistic data.

On BIRD, a well-known text-to-SQL benchmark built on real databases, humans score about 93% execution accuracy. The best systems on the leaderboard were around 83% as of October 2026. That's already a real gap. Then there's Spider 2.0, which uses real enterprise-style databases, often with over 1,000 columns. In the Spider 2.0 paper (Lei et al., ICLR 2025), an o1-preview-based agent solved only 21.3% of the tasks.

So the model will make mistakes. My job isn't to make it perfect. My job is to make sure that when it's wrong, it's wrong safely and visibly.

The setup

Three pieces:

• DuckDB: an in-process analytical database. It reads CSVs directly and doesn't need a server. Good for a student laptop.
• sqlglot: a Python SQL parser. It turns a query into a syntax tree so I can inspect what it actually does instead of guessing with regex.
• Any LLM with an OpenAI-compatible API: hosted or local, it doesn't matter. The model is the part I trust least.

pip install duckdb sqlglot openai

Step 1: Load the data, then lock the door

import duckdb

con = duckdb.connect()  # in-memory database
con.execute("CREATE TABLE sales AS SELECT * FROM read_csv_auto('sales.csv')")

# From here on, no query can read or write files on my machine.
con.execute("SET enable_external_access = false")
con.execute("SET lock_configuration = true")

This is my favorite part, because it doesn't depend on the LLM behaving. With external access turned off, functions like read_csv can't reach files or URLs anymore. Locking the configuration means a sneaky generated query can't turn that setting back on. I load my data first and lock everything after.

(DuckDB's own security docs recommend OS-level sandboxing for truly untrusted SQL, so treat this as one layer, not the whole wall.)

If you're using a database file instead of memory, also open it with duckdb.connect("data.duckdb", read_only=True).

Step 2: Give the model the schema, not the data

The model can't guess column names it has never seen. So I send it the table structure, and only that:

def get_schema(con) -> str:
    rows = con.execute("""
        SELECT table_name, column_name, data_type
        FROM information_schema.columns
        ORDER BY table_name, ordinal_position
    """).fetchall()
    return "\n".join(f"{t}.{c} ({d})" for t, c, d in rows)

No rows, no customer data, just names and types. Better for privacy, and it keeps the prompt small.

Step 3: Ask the model for SQL

from openai import OpenAI

client = OpenAI()   # point base_url at any OpenAI-compatible endpoint
MODEL = "your-model-name"

def ask_llm(question: str, schema: str, error: str | None = None) -> str:
    prompt = f"Schema:\n{schema}\n\nQuestion: {question}\n"
    if error:
        prompt += f"\nYour last query failed with: {error}\nFix it."
    resp = client.chat.completions.create(
        model=MODEL,
        temperature=0,
        messages=[
            {"role": "system", "content": "You write one DuckDB SELECT query. "
             "Return only SQL, no explanation."},
            {"role": "user", "content": prompt},
        ],
    )
    sql = resp.choices[0].message.content.strip()
    return sql.removeprefix("```sql").removesuffix("```").strip()

Notice the system prompt says "only SELECT." That's a polite request, not a security feature. Models don't always follow instructions, which is exactly why Step 4 exists.

Step 4: The guard

This is the heart of it. Before anything touches the database, the SQL gets parsed and checked:

import sqlglot
from sqlglot import exp
from sqlglot.errors import ParseError

ALLOWED_TABLES = {"sales"}
READ_ONLY_ROOTS = (exp.Select, exp.Union, exp.Intersect, exp.Except)
WRITE_NODES = (exp.Insert, exp.Update, exp.Delete, exp.Drop, exp.Create)

def guard(sql: str) -> str:
    try:
        statements = [s for s in sqlglot.parse(sql, read="duckdb") if s is not None]
    except ParseError as e:
        raise ValueError(f"Could not parse SQL: {e}")

    if len(statements) != 1:
        raise ValueError("Exactly one statement allowed.")
    tree = statements[0]

    if not isinstance(tree, READ_ONLY_ROOTS):
        raise ValueError(f"Only read-only queries allowed, got {type(tree).__name__}.")
    if any(True for _ in tree.find_all(*WRITE_NODES)):
        raise ValueError("Write operation found inside the query.")

    cte_names = {c.alias_or_name.lower() for c in tree.find_all(exp.CTE)}
    for table in tree.find_all(exp.Table):
        name = table.name.lower()
        if not name:
            raise ValueError("Table functions are not allowed.")
        if name not in ALLOWED_TABLES | cte_names:
            raise ValueError(f"Unknown table: {name}")

    if isinstance(tree, exp.Select) and not tree.args.get("limit"):
        tree = tree.limit(200)

    return tree.sql(dialect="duckdb")

A few things I learned while writing this:

• Parse, don't pattern-match. Checking whether the string contains "DROP" breaks the moment someone adds a comment or changes the casing. A syntax tree doesn't care.
• Run the checked version, not the original string. I return tree.sql(), the query I actually inspected, not whatever text the model sent back.
• sqlglot isn't a security tool. Its own docs say the parser is intentionally lenient. It's a transpiler, not a validator. That's why the DuckDB lockdown in Step 1 exists. If one layer misses something, the next one catches it.

Step 5: One retry, then be honest

def answer(question: str):
    schema = get_schema(con)
    error = None
    for attempt in range(2):
        sql = ask_llm(question, schema, error)
        try:
            safe_sql = guard(sql)
            result = con.execute(safe_sql).fetchdf()
            return safe_sql, result
        except Exception as e:
            error = str(e)
    return None, f"Couldn't answer that safely. Last error: {error}"

sql, result = answer("Which region had the highest total revenue last quarter?")
print(sql)
print(result)

If the query fails (say, because of a made-up column), the error goes back to the model once. If it fails again, the tool says so out loud. I'd much rather DataCopilot say "I couldn't do that" than quietly return junk.

I also always show the SQL next to the answer. That's not just for debugging, it's for trust. If someone asks about "last quarter," they should be able to see which dates the model actually picked.

What this still doesn't solve

Being honest here, because this is where the real work is:

• Safe isn't the same as correct. The guard stops a DROP TABLE. It does nothing about a query that happily averages the wrong column. That's a correctness problem, and it needs evaluation: a set of questions with known answers, checked by actually running both queries and comparing results.
• Business meaning lives outside the schema. What counts as an "active customer"? The schema doesn't know, and neither does the model. You have to tell it.
• Big schemas get messy fast. One table is easy. Real company databases with hundreds of columns are where accuracy falls apart.

What's next

Part 2 will be about evaluation: building a small test set for my own data and measuring how often the answers are actually right, not just whether the SQL runs.

After that, I want to push toward something closer to my research interests. Not just "what happened in the data," but "what would change if we did X." That's where causal inference and uplift modeling come in, and honestly, it's the part of DataCopilot I'm most excited about.

If you've built something like this, I'd love to hear how you handle correctness, because that's the part I'm still figuring out. Drop a comment, or connect with me through the links on my portfolio.

And if you're a student building your first AI project: put the guardrails in on day one. It's way easier than adding them after something breaks.

Rust was never supposed to be an AI language

Rust came out of Mozilla, built for safe systems programming: memory safety without a garbage collector, no data races, zero-cost abstractions. None of that was aimed at AI. AI wasn't even a consideration when the language was designed. But it turns out those exact properties are precisely what you want underneath a model that's serving real traffic at 3am, not just running in a notebook.

Python still owns research and model development, and that's not changing. PyTorch, JAX, the whole ecosystem lives there. What's changing is everything around the model.

Where it's already running under you right now

A few examples that surprised me when I actually went looking:

Hugging Face's tokenizers are written in Rust. Every time you tokenize text through the transformers library, there's a good chance Rust is doing the actual work under a Python wrapper. Same story with Pydantic's core validation layer.

Polars is replacing Pandas for a lot of people. It's a dataframe library built in Rust, and for large datasets the ETL speed difference over Pandas is significant, not a marginal improvement.

Candle exists now. Hugging Face built a pure-Rust ML framework that runs on CPU, CUDA, and Metal. There's also Burn, aimed more at embedded and edge deployment, and tch-rs, which gives you PyTorch-like bindings directly in Rust.

Agent infrastructure is shifting hard toward Rust. This is the one that actually made me sit up. Early this year, a clear pattern showed up in GitHub's fastest-growing AI agent repos: agent runtimes, CLI tools, and sandboxes increasingly being built in Rust instead of Python or TypeScript. When your agent is running with real permissions on someone's machine, memory safety stops being a nice-to-have.

Why the split makes sense

The honest way to think about it: Python is the scripting layer, Rust is becoming the runtime layer. You prototype and train in Python because iteration speed matters more than raw performance at that stage. But the layer in front of your model, the gateway, the router, the streaming layer, the sandboxing around an agent, that's a different job. That's where Rust's lack of garbage-collector pauses actually shows up as flatter tail latency under load, and where memory safety matters because that code is handling untrusted input at scale.

Rust isn't replacing Python here. It's replacing the C and C++ that used to sit underneath Python libraries anyway, just with something that doesn't crash the same way.

What learning it has actually been like so far

I'm early, so take this as a first-impressions report, not an expert take. The ownership and borrowing model is the thing everyone warns you about, and they're right to. Coming from Python, where you never think about who "owns" a value, having the compiler refuse to build until you've proven your memory usage is sound feels restrictive at first. It is restrictive. That's the whole point.

What's actually clicking for me is realizing how many bugs I've shipped in Python that Rust's compiler would have caught before the code even ran. Not exceptions I handled gracefully. Bugs I didn't know existed until production told me.

I don't think every ML engineer needs to become a Rust developer. But I do think the "it's just a systems language, not relevant to AI work" take is already out of date, and I'd rather be the person who noticed that a year early than a year late.

If you're also picking up Rust coming from a Python/ML background, I'd like to compare notes. Find me on X, or check out what I've been building at github.com/mritunjaySharma05.

What's actually new

Astra came out as a limited preview on September 3rd, with wider paid access the next day. It's better at coding, research, computer use, and multi-step tasks, sure, fine, every model says that now. The two things that actually got my attention:

Computer-use is almost 2x faster. And here's the part I liked: the speedup isn't locked to Astra. OpenAI rolled the same harness optimizations into GPT-5.6 Sol too, and apparently that alone got a ~60% speed boost. Which tells you something. A chunk of what feels like "the new model is smarter" is really just "someone fixed the plumbing." Not glamorous, but it's the kind of improvement that actually shows up when you're using the thing daily instead of watching a demo.

Codex stopped just summarizing its way out of long tasks. This one's personal for me. Every agent builder knows the pain: your context window fills up mid-debug or mid-refactor, and the model compacts (a fancy word for "summarizes and hopes nothing important got dropped"). Astra apparently handles this differently. It preserves and retrieves context instead of just compressing it into mush. If you've ever watched an agent completely lose the plot 40 minutes into a long task, you know exactly why this matters.

Why the rollout is so slow

This isn't a "everyone gets it today" launch. It's rolling out first to a small set of orgs in OpenAI's cybersecurity program (Daybreak), and general access is trickling out after that. They've also locked down some of Astra's more advanced cybersecurity capabilities specifically. Honestly, given that OpenAI reportedly delayed this release after the Hugging Face hacking incident earlier this year, the caution tracks. I don't think that's PR spin. That's a company that got spooked and is being visibly careful this time.

The part that actually connects to my own project

Here's why I'm writing about this at all instead of just doom-scrolling past it: I hit this exact problem building my FastMCP research agent. Long-running research task, lots of intermediate state (search results, extracted snippets, running notes), and if anything failed halfway through, I didn't want to lose all of it. My fix was boring but effective: write every step incrementally to SQLite instead of trusting it all to stay alive in context.

That works. But it's also a workaround for a limitation that apparently OpenAI is now trying to fix at the model level instead of leaving it to whoever's building on top. And that's the actual lesson I'm taking from this release, more than any benchmark number: every time a frontier model ships, it's worth asking not "is this smarter" but "which of my current hacks does this quietly make pointless."

Where I actually land on this

I'm not going to sit here and verify OpenAI's own "most aligned model yet" claim. That's their marketing, not something I can check from where I'm sitting. But the specific, boring, checkable stuff (the phased Daybreak rollout, the retroactive speed bump to an older model, the shift away from pure summarization for long-context work) is the stuff that actually matters once you're building with these things day to day, not just reading about them.

If you're deep in agent-building right now like I am, I wouldn't rush to rewrite anything. Astra isn't even broadly available yet. But it's a good moment to notice which of your workarounds exist purely because the model couldn't do better, and leave room in your design to rip them out later when it can.

If you're building with agents or RAG and running into similar walls, I'd genuinely like to hear about it. Find me on X, or take a look at my FastMCP research agent on GitHub.

This post walks through how I built an end-to-end, fully automated data platform on Snowflake to answer that question — from raw API ingestion to a live dashboard — along with the platform decisions and bugs that shaped the final design.

The problem, and why Snowflake

Two public APIs feed this pipeline: USGS for live earthquake data, and Open-Meteo for weather conditions at each event's coordinates. The tricky part isn't fetching the data — it's modeling it well and keeping it fresh without babysitting it.

I evaluated three platforms before settling on Snowflake: Databricks, Microsoft Fabric, and Snowflake itself. Snowflake won for a few concrete reasons:

• No cluster management. Databricks needs you to think about compute clusters; Snowflake's warehouses abstract that away entirely.

• Native JSON handling. USGS earthquake data comes back as deeply nested GeoJSON. Snowflake's VARIANT type plus FLATTEN() let me unnest that directly in SQL — no separate Spark/PySpark layer required.

• Built-in scheduling. Snowflake Tasks handle automation natively. No need to stand up an external orchestrator like Data Factory.

• Native dashboarding. Streamlit in Snowflake ships visualization inside the platform itself, so I didn't need to wire up Power BI or Tableau separately.

For a project this size, every one of those meant one less moving part to maintain.

The architecture

At a high level: a Python ingestion script pulls from both APIs and lands raw JSON locally. From there, two paths run in parallel —

1. One loads the flattened data into a normalized (3NF) staging schema

2. The other loads the original nested JSON as a VARIANT column, preserved as-is

A stored procedure then combines both to populate a star schema, and the whole thing runs automatically every hour via a Snowflake Task.

I modeled the data twice — once normalized, once raw — quite deliberately. The 3NF staging tables (EARTHQUAKES and WEATHER, joined on event_id) give me a clean, query-friendly layer for day-to-day analytics. The raw VARIANT copy preserves the original nested structure so I can always go back to source truth and reconstruct anything I might have flattened incorrectly the first time. It's a small amount of extra storage for a lot of downstream flexibility.

Handling data that changes after the fact

Here's something I didn't expect going in: USGS revises earthquake records after publishing them. A magnitude gets updated as more sensor data comes in, and the same event_id shows up again with new values.

I handled this with a MERGE (upsert) on event_id: if the event already exists, it only updates when the incoming record's updated_epoch_ms is newer than what's stored. Otherwise, it inserts a new row. Weather observations, by contrast, are point-in-time and never revised at the source so those are simple inserts, no merge logic needed. Small distinction, but it mattered for correctness.

The star schema

The final analytical layer is a proper star schema:

• Fact table: FACT_EARTHQUAKE_WEATHER — one row per earthquake, joined with its weather reading

• Dimensions: DIM_DATE, DIM_LOCATION, DIM_ALERT

The interesting part is how it gets built. The stored procedure TRANSFORM_TO_STAR_SCHEMA() uses LATERAL FLATTEN(input => payload:features) to unnest the raw GeoJSON directly from the VARIANT column, and every dimension plus the fact table uses MERGE to avoid duplicate rows across repeated hourly runs. The whole flatten-and-load process is one callable unit which makes the hourly automation trivial to wire up.

Three things that actually broke

Every project has a gap between "the architecture diagram" and "what actually happened." Here's what tripped me up:

1. Schema context inside stored procedures. Unqualified table names inside the stored procedure initially failed, because session schema context isn't guaranteed to be set at call time. The fix was straightforward once I understood it — fully qualify every table reference as database.schema.table — but it cost me a confusing debugging session before I traced it back to that.

2. Data freshness mismatch. My raw-VARIANT loader and my staging loader both pull from the USGS feed independently. When their runs didn't land on the exact same set of earthquakes, weather enrichment would show up as NULL for some events. This wasn't a bug in my join logic — it was a timing issue between two independent ingestion paths. I resolved it by having the automated hourly Task run the transform immediately after ingestion, keeping both paths tightly synced.

3. Snowsight Dashboards got deprecated mid-project. Genuinely — Snowflake deprecated the legacy Dashboards feature while I was building this (new dashboard creation disabled April 2026, full removal beginning June 2026). I pivoted to Streamlit in Snowflake, the officially supported replacement, which turned out better anyway since it runs natively inside the platform with no extra service to manage.

Automation and data quality

Two Snowflake Tasks run the show:

The data quality checks cover the basics you'd want in any pipeline handling real-world data: magnitude range validation, non-negative depth, valid lat/lon bounds, duplicate event_id detection, orphaned foreign keys in the fact table, and plausible weather value ranges. None of this is glamorous, but it's the difference between a dashboard you can trust and one you can't.

The dashboard

streamlit_app.py — running as Streamlit in Snowflake — surfaces:

• KPI tiles (total earthquakes, average/max magnitude, weather-risk event count)

• A magnitude distribution chart

• Activity-by-hour trends

• A geographic map of events

• A rescue-risk table — the actual core use case: which recent earthquakes are paired with weather that could complicate a response

• A top-10 significant events table

That rescue-risk table is really the point of the whole pipeline. Everything upstream — the ingestion, the two parallel data models, the star schema, the hourly automation — exists to make that one table accurate and current.

What I'd take into the next project

A few things I'll carry forward:

• Model raw data twice if you can afford the storage. Having the untouched VARIANT copy alongside the normalized tables saved me more than once when I needed to double-check whether a bug was in my flattening logic or in the source data itself.

• Timing bugs disguise themselves as logic bugs. The NULL weather values looked like a join problem at first. It wasn't — it was two independent processes running out of sync. Worth checking timing before you start rewriting queries.

• Plan for your platform to change under you. The Snowsight Dashboards deprecation wasn't something I could have predicted, but building on Snowflake's currently-recommended path (Streamlit in Snowflake) rather than a soon-to-be-legacy feature would have avoided the pivot entirely. Lesson: check a feature's roadmap status before committing to it mid-project.

Full code, SQL, and architecture diagrams are on GitHub: earthquake-crisis-pipeline

If you're working on anything similar crisis-informatics pipelines, Snowflake automation, or just fighting with nested JSON I'd love to hear about it.