{"id":4244,"date":"2026-10-16T08:01:08","date_gmt":"2026-10-16T08:01:08","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=4244"},"modified":"2026-10-16T08:01:11","modified_gmt":"2026-10-16T08:01:11","slug":"great-expectations-for-data-validation-in-dataops-a-practical-guide","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/great-expectations-for-data-validation-in-dataops-a-practical-guide\/","title":{"rendered":"Great Expectations for Data Validation in DataOps: A Practical Guide"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

Your pipeline ran green all night. Every job succeeded. Then your phone rings at 8 a.m. The sales dashboard shows revenue dropped 70% overnight. Nothing crashed. Nothing failed. The data simply arrived broken, and every downstream report quietly spread the damage.<\/p>\n\n\n\n

This is the daily pain of every data team. Pipelines tell you the code ran. They do not tell you the data makes sense. A trailing zero dropped here, a null explosion there, and your “trusted” dashboard becomes a liability.<\/p>\n\n\n\n

Great Expectations fixes this. It is an open-source Python tool that tests your data the same way developers test code. You write simple rules, called expectations, and your pipeline proves its output is correct before anyone sees it. To learn DataOps skills end to end, including hands-on data validation practice, explore the courses at DataOpsSchool.com<\/a>.<\/p>\n\n\n\n

In this guide, we break down how Great Expectations works, why it sits at the heart of DataOps, where teams fail with it, and how to roll it out in ten clear steps.<\/p>\n\n\n\n

What Is Great Expectations?<\/h2>\n\n\n\n

Great Expectations is a free, open-source data testing framework written in Python. Think of it as unit testing for your data. Instead of testing functions, you test tables, columns, and values.<\/p>\n\n\n\n

An expectation is a simple statement about your data that must be true. For example: “I expect the order_id<\/code> column to be unique.” Or: “I expect order_total<\/code> to be between 0 and 1 million.” Each expectation returns a pass or a fail. There is no gray area.<\/p>\n\n\n\n

The tool connects to almost any data source. It works with SQL databases, Snowflake, BigQuery, Databricks, Spark, Pandas files, and cloud storage. You describe your data once, and the checks follow it everywhere.<\/p>\n\n\n\n

Here are examples of expectations in plain English:<\/p>\n\n\n\n

    \n
  • expect_column_values_to_not_be_null<\/strong> \u2014 every row must have a value here.<\/li>\n\n\n\n
  • expect_column_values_to_be_unique<\/strong> \u2014 no duplicates allowed.<\/li>\n\n\n\n
  • expect_column_values_to_match_regex<\/strong> \u2014 values must follow a fixed pattern, like a phone number.<\/li>\n\n\n\n
  • expect_column_values_to_be_between<\/strong> \u2014 numbers must stay inside a safe range.<\/li>\n\n\n\n
  • expect_table_row_count_to_be_between<\/strong> \u2014 the table cannot be empty or absurdly large.<\/li>\n<\/ul>\n\n\n\n

    Because expectations read like English, business users can understand them too. Your data tests stop being a developer secret. They become shared team knowledge.<\/p>\n\n\n\n

    Why Data Validation Is the Heart of DataOps<\/h2>\n\n\n\n

    DataOps borrows the best ideas from software DevOps. You build pipelines with version control. You test. You deploy. You monitor. But software has one advantage data never gets: code behaves the same every run. Data does not.<\/p>\n\n\n\n

    Your data changes every single day. New records arrive. Sources change their formats. Vendors send odd files. A pipeline that worked yesterday can break today without a single code change. That is why testing the data itself, not just the code, matters so much.<\/p>\n\n\n\n

    The cost of skipping this step is huge. Gartner research puts the average cost of poor data quality at about $12.9 million per year for a single organization. Bad data also wastes people’s time. In many surveys, data teams report that a large share of their week goes to fixing data issues and answering “is this number right?” questions.<\/p>\n\n\n\n

    Data validation closes that gap. When Great Expectations runs inside your pipeline, bad data gets stopped at the door. Your team stops reacting and starts trusting.<\/p>\n\n\n\n

    Validation also protects your DataOps metrics. Cycle time drops because failures surface early. Rework drops because broken data never reaches reports. Trust rises because every dataset carries proof that it passed its checks.<\/p>\n\n\n\n

    So validation is not a nice extra. It is the control layer that makes DataOps real.<\/p>\n\n\n\n

    How Great Expectations Works: The Core Building Blocks<\/h2>\n\n\n\n

    Great Expectations uses four main pieces. Once you understand them, the whole tool clicks into place.<\/p>\n\n\n\n

    The first piece is the Expectation Suite<\/strong>. This is a saved collection of expectations for one dataset. Think of it as a checklist for your table. Your orders<\/code> table might have 25 checks covering nulls, ranges, formats, and row counts. Suites are stored as JSON or YAML files, so you can version them in Git like code.<\/p>\n\n\n\n

    The second piece is the Checkpoint<\/strong>. A checkpoint runs one suite against one batch of data. It connects three things: the data, the suite, and the action to take afterward. Actions can include updating Data Docs, sending Slack alerts, or storing validation results.<\/p>\n\n\n\n

    The third piece is Validation Results<\/strong>. Every run produces a record of each expectation, its success, and a result summary. Failed checks show you the exact rows and values that broke the rules, so debugging is fast.<\/p>\n\n\n\n

    The fourth piece is Data Docs<\/strong>, which we cover next.<\/p>\n\n\n\n

    Here is how the four pieces connect:<\/p>\n\n\n\n

    Building Block<\/th>What It Does<\/th>Where It Lives<\/th><\/tr>
    Expectation Suite<\/td>Stores all checks for one dataset<\/td>YAML\/JSON files in Git<\/td><\/tr>
    Checkpoint<\/td>Runs the suite on a schedule or trigger<\/td>Pipeline task (Airflow, dbt, CI)<\/td><\/tr>
    Validation Results<\/td>Records every pass and fail with details<\/td>Results store or database<\/td><\/tr>
    Data Docs<\/td>Shows results as a clean HTML site<\/td>Hosted page or local folder<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Data Docs: Your Pipeline’s Report Card<\/h2>\n\n\n\n

    Data Docs is the feature that wins people over. Every validation run updates an HTML website automatically. That site shows every suite, every expectation, and every result in clean, readable pages.<\/p>\n\n\n\n

    No SQL digging. No log scrolling. You open a page and see a green check or a red flag for each rule. Failed checks include sample values, so you see the bad data right away.<\/p>\n\n\n\n

    Data Docs also solves a trust problem. Analysts and business users can open the docs and see, in plain terms, what checks the data passed. When someone asks “can we trust yesterday’s numbers?”, you send a link instead of writing a paragraph.<\/p>\n\n\n\n

    Stakeholders who see validation evidence stop asking for manual sign-off. That is a real DataOps win: trust that scales without meetings.<\/p>\n\n\n\n

    Keep one caution. Data Docs can contain sample values from your data. If you work with sensitive fields, sample only safe columns or restrict access to the docs site.<\/p>\n\n\n\n

    Where Great Expectations Fits in Your DataOps Pipeline<\/h2>\n\n\n\n

    Great Expectations is not another system to babysit. It plugs into the tools you already run.<\/p>\n\n\n\n

    In Airflow<\/strong>, you add a validation task after each load. If the checkpoint fails, the task fails, and the DAG stops bad data from moving downstream. You can also run validations as a quality gate between staging and production tables.<\/p>\n\n\n\n

    With dbt<\/strong>, you can run checkpoints after dbt build<\/code> completes. Some teams pair dbt’s built-in tests with Great Expectations suites for deeper statistical checks that dbt tests do not cover.<\/p>\n\n\n\n

    In Databricks and Spark<\/strong>, Great Expectations validates DataFrames before they land in curated tables. You catch issues inside the job, not after.<\/p>\n\n\n\n

    In CI\/CD<\/strong>, teams run suites on sample datasets during pull requests. A change that breaks a data contract gets blocked before merge, just like a failing unit test.<\/p>\n\n\n\n

    The pattern is always the same: validate at the moment data moves. Check raw data at ingest. Check staged data after cleaning. Check curated data before publishing. Each gate keeps damage contained to one layer.<\/p>\n\n\n\n

    Great Expectations vs Other Validation Tools<\/h2>\n\n\n\n

    Great Expectations is popular, but it is not the only option. Here is an honest comparison of the leading tools.<\/p>\n\n\n\n

    Tool<\/th>Best For<\/th>Language<\/th>Learning Curve<\/th>Docs Output<\/th><\/tr>
    Great Expectations<\/td>Broad pipeline validation, rich Data Docs<\/td>Python<\/td>Medium<\/td>Excellent HTML Data Docs<\/td><\/tr>
    dbt tests<\/td>Teams already using dbt<\/td>SQL<\/td>Low<\/td>Built into dbt Cloud<\/td><\/tr>
    Soda Core<\/td>SQL-friendly checks and alerts<\/td>YAML\/SQL<\/td>Low to medium<\/td>Cloud dashboards<\/td><\/tr>
    Deequ<\/td>Large-scale Spark jobs on AWS<\/td>Scala\/Python<\/td>Medium to high<\/td>Metrics repository<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    Manual checks versus Great Expectations tells an even sharper story:<\/p>\n\n\n\n

    Factor<\/th>Manual SQL Checks<\/th>Great Expectations<\/th><\/tr>
    Speed of setup<\/td>Fast for one query<\/td>Fast for one suite<\/td><\/tr>
    Reuse<\/td>Copy-paste every time<\/td>Suites reused across tables<\/td><\/tr>
    Alerting<\/td>Manual review<\/td>Automatic on failure<\/td><\/tr>
    Documentation<\/td>Stale wiki pages<\/td>Living Data Docs<\/td><\/tr>
    Scaling<\/td>Breaks with team size<\/td>Scales with Git and CI\/CD<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

    The right answer for most teams is not either\/or. Many DataOps teams run dbt tests for simple schema checks and Great Expectations for range, distribution, and cross-table rules.<\/p>\n\n\n\n

    Real-World Stories: When Bad Data Hit Hard<\/h2>\n\n\n\n

    These ten scenarios come from patterns seen again and again in real data teams. Each one shows how validation would have changed the ending.<\/p>\n\n\n\n

      \n
    • The 70% revenue drop that never happened.<\/strong> A SaaS company’s finance dashboard showed a huge revenue crash on Monday. The cause was a partner API that switched a date format overnight. A single regex expectation on the date column would have caught it at ingest, before the CFO saw it.<\/li>\n\n\n\n
    • Duplicate customers double the invoice list.<\/strong> A retail firm loaded the same customer file twice because a scheduler retried silently. Marketing sent two offers to thousands of customers. A uniqueness expectation on the customer ID would have failed the second load.<\/li>\n\n\n\n
    • The null avalanche.<\/strong> A telecom analytics team saw churn scores collapse for 40% of users. An upstream team renamed a column, and the pipeline read it as null. A “not null” expectation, plus a check that null rates stay under 1%, would have raised a red flag within minutes.<\/li>\n\n\n\n
    • Currency confusion in a global report.<\/strong> A travel company summed bookings in mixed currencies without conversion, inflating revenue by 30%. An expectation that currency codes match an allowed set, plus range checks on totals, would have forced a fix before publication.<\/li>\n\n\n\n
    • Timezone drift breaks weekly reports.<\/strong> A fintech startup stored timestamps in local server time. Daylight saving shifted an hour of transactions into the wrong day, breaking daily reconciliation. A timestamp format expectation and row-count window checks would have flagged the anomaly.<\/li>\n\n\n\n
    • The empty file that “succeeded.”<\/strong> A logistics pipeline read a zero-row extract for three days and showed blank KPIs. No job failed because the file technically arrived. A minimum row-count expectation (expect_table_row_count_to_be_between) makes empty loads impossible to miss.<\/li>\n\n\n\n
    • Free-text fields poison a machine learning model.<\/strong> A health tech firm fed a state column containing “CA”, “Ca”, “California”, and “CALIFORNIA”. Model accuracy dropped and nobody knew why. A value-set expectation forces clean, consistent categories from day one.<\/li>\n\n\n\n
    • The schema change nobody announced.<\/strong> A vendor renamed 12 columns in a feed without notice. The pipeline loaded nulls into renamed fields for a full week. A suite that checks column names and types fails loudly the moment a schema shifts.<\/li>\n\n\n\n
    • Stale data in an executive dashboard.<\/strong> An insurance company’s exec dashboard showed last week’s data because a loader hung silently. Row-count expectations that compare today’s load to yesterday’s catch stalled feeds fast.<\/li>\n\n\n\n
    • The typo that cost millions in targeting.<\/strong> An e-commerce team filtered orders by country code “IN” but a source sent “IN ” with a trailing space. Indian orders vanished from regional reports for a month. A trim and value-set expectation surfaces such bugs on the first run.<\/li>\n<\/ul>\n\n\n\n

      The pattern repeats: silent data errors cause loud business pain. Validation turns silent breakage into a loud, early failure.<\/p>\n\n\n\n

      Common Mistakes and Pitfalls<\/h2>\n\n\n\n

      Most Great Expectations struggles come from how teams adopt it, not from the tool itself. Here are the ten most frequent mistakes, each with the fix.<\/p>\n\n\n\n

        \n
      • Writing 200 expectations on day one.<\/strong> Teams drown in failed checks and disable the tool in week two. Fix: start with 10 to 15 critical checks per table. Grow only after each one earns its place.<\/li>\n\n\n\n
      • Testing everything at the end of the pipeline.<\/strong> By the time curated data fails, hours of compute are wasted. Fix: add gates at ingest, staging, and publish so failures happen early and cheap.<\/li>\n\n\n\n
      • Making every failure a pager alert.<\/strong> If low-value checks wake people at night, they mute all alerts. Fix: route critical checks to pages and warnings to Slack digests. Severity matters.<\/li>\n\n\n\n
      • Never reviewing expectation health.<\/strong> Stale suites drift out of touch with real data, and teams trust them less. Fix: review suites monthly. Remove dead checks. Add checks for new failure patterns.<\/li>\n\n\n\n
      • Hard-coding ranges forever.<\/strong> A fixed range like 0 to 10,000 breaks when the business grows. Fix: use mostly-based statistical expectations that learn normal ranges from recent data.<\/li>\n\n\n\n
      • Ignoring Data Docs.<\/strong> Teams generate docs but never share them, so business users keep asking for manual checks. Fix: publish Data Docs and link them from your dashboards. Let proof do the talking.<\/li>\n\n\n\n
      • Treating suites as private code.<\/strong> When only one engineer owns the checks, coverage dies when that person leaves. Fix: keep suites in Git, review them like code, and let analysts propose new expectations.<\/li>\n\n\n\n
      • Skipping data source testing.<\/strong> Teams validate warehouse data but trust raw files blindly. Fix: run suites against landing zones and APIs too. Bad data caught at the source never spreads.<\/li>\n\n\n\n
      • Failing without context.<\/strong> A checkpoint that fails with no detail forces hours of digging. Fix: always attach result summaries and Data Docs links to your alert messages.<\/li>\n\n\n\n
      • Expecting Great Expectations to fix data.<\/strong> It validates; it does not repair. Fix: pair failed checks with clear remediation tasks. Decide who fixes, how fast, and what blocks downstream publishing.<\/li>\n<\/ul>\n\n\n\n

        Step-by-Step Implementation Roadmap<\/h2>\n\n\n\n

        Follow these ten steps in order. Each one builds on the last, and the whole rollout can happen in two to four weeks for a small team.<\/p>\n\n\n\n

          \n
        1. Pick one painful pipeline.<\/strong> Choose a dataset that has hurt the business before, ideally one feeding a visible dashboard. Early wins build support for a wider rollout.<\/li>\n\n\n\n
        2. Profile the data first.<\/strong> Run the built-in profiler on a sample. It suggests expectations based on real values, so you start from evidence instead of guesses.<\/li>\n\n\n\n
        3. Create your first suite.<\/strong> Keep it to 10 to 15 checks covering not-null, uniqueness, row counts, and ranges on key columns. Save the suite in Git with the pipeline code.<\/li>\n\n\n\n
        4. Run a checkpoint manually.<\/strong> Execute the suite against a batch of live data. Expect failures on the first run, and use them to learn what “normal” truly looks like.<\/li>\n\n\n\n
        5. Refine and re-run.<\/strong> Fix expectations that contradict real data, and tighten ones that expose genuine problems. This loop, usually two or three cycles, sets your quality baseline.<\/li>\n\n\n\n
        6. Generate Data Docs.<\/strong> Publish the docs site and share it with analysts and business owners. Watch their questions shift from “is this right?” to “which checks passed?”<\/li>\n\n\n\n
        7. Wire the checkpoint into your orchestrator.<\/strong> Add the validation task to Airflow, dbt, or your scheduler right after the load step. Make failure block downstream publishing.<\/li>\n\n\n\n
        8. Set up alerting with severity.<\/strong> Route critical failures to on-call pages and soft warnings to chat channels. Test both paths with a forced failure before you rely on them.<\/li>\n\n\n\n
        9. Expand suite by suite.<\/strong> Apply the same pattern to the next table, then the next feed. Reuse proven expectations across similar datasets to move faster each time.<\/li>\n\n\n\n
        10. Review and improve monthly.<\/strong> Check failure history, retire stale checks, and add rules for new failure modes. Treat suite quality like code quality, with reviews and owners.<\/li>\n<\/ol>\n\n\n\n

          Frequently Asked Questions<\/h2>\n\n\n\n

          1. What is Great Expectations in DataOps?<\/h3>\n\n\n\n

          Great Expectations is an open-source Python framework that tests data inside pipelines. In DataOps, it acts as an automated quality gate that checks tables, columns, and values against rules called expectations, so bad data gets stopped before it reaches reports.<\/p>\n\n\n\n

          2. Is Great Expectations free to use?<\/h3>\n\n\n\n

          Yes. Great Expectations is open source and free to download and run. There is also a paid cloud option with extra management features, but the core framework, including Expectation Suites, Checkpoints, and Data Docs, costs nothing.<\/p>\n\n\n\n

          3. Which data sources does Great Expectations support?<\/h3>\n\n\n\n

          It supports most popular sources, including PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, Databricks, Spark, Pandas, and files on cloud storage like S3. New sources are added through its fluent datasource configuration system.<\/p>\n\n\n\n

          4. How is Great Expectations different from dbt tests?<\/h3>\n\n\n\n

          dbt tests are simple SQL checks that live inside dbt models, great for schema and uniqueness rules. Great Expectations offers deeper checks like statistical ranges, distribution shifts, and cross-column logic, plus rich HTML documentation. Many teams use both together.<\/p>\n\n\n\n

          5. Can Great Expectations run inside Airflow?<\/h3>\n\n\n\n

          Yes. You can run a Great Expectations checkpoint as an Airflow task right after a load step. If validation fails, the task fails, and the pipeline stops bad data from moving downstream.<\/p>\n\n\n\n

          6. What are Data Docs in Great Expectations?<\/h3>\n\n\n\n

          Data Docs are auto-generated HTML pages that show every expectation, suite, and validation result in a clean, readable format. They serve as living documentation and proof of data quality that non-technical stakeholders can review.<\/p>\n\n\n\n

          7. How many expectations should a team start with?<\/h3>\n\n\n\n

          Start with 10 to 15 checks per critical table. Cover not-null rules, uniqueness, row counts, and value ranges on key columns. Add more only when each existing check has proven its value in real runs.<\/p>\n\n\n\n

          8. Does Great Expectations slow down pipelines?<\/h3>\n\n\n\n

          Validation adds some runtime, usually seconds per checkpoint on well-designed suites. Sampling large tables and validating only key columns keep the cost low while still catching most data issues.<\/p>\n\n\n\n

          9. Can Great Expectations fix bad data automatically?<\/h3>\n\n\n\n
            \n
          1. It detects and reports data problems; it does not repair them. Your team decides the fix, whether that is reloading, cleaning, or blocking the load, and the failed expectation tells you exactly where the problem lives.<\/li>\n<\/ol>\n\n\n\n

            10. Is Great Expectations good for machine learning data validation?<\/h3>\n\n\n\n

            Yes. It is widely used to validate training and feature data. Expectations for value ranges, category sets, null rates, and distribution shifts catch feature problems before they poison a model, and suites can run before every training job.<\/p>\n\n\n\n

            \n\n\n\n

            Final Recommendation and Next Steps<\/h2>\n\n\n\n

            Start small, but start this week. Pick the one pipeline that has embarrassed your team most, and put 10 expectations on it. That single step catches the majority of silent failures, and it gives you a story that wins budget for the rest.<\/p>\n\n\n\n

            Treat validation as a product, not a project. Suites need owners, reviews, and a monthly check-in. The teams that keep Data Docs current and alerting sharp are the ones whose dashboards people trust without a second thought. The teams that “finish” validation are the ones who quietly drift back into 8 a.m. fire drills.<\/p>\n\n\n\n

            Your next move is concrete: install Great Expectations, profile your most-watched table, and ship one checkpoint. In one afternoon you can turn your quietest fear, broken data, into your loudest early warning system.<\/p>\n","protected":false},"excerpt":{"rendered":"

            Your pipeline ran green all night. Every job succeeded. Then your phone rings at 8 a.m. The sales dashboard shows revenue dropped 70% overnight. Nothing crashed. Nothing… <\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4244","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4244"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4244\/revisions"}],"predecessor-version":[{"id":4246,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4244\/revisions\/4246"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}