D497 Data Wrangling help

Diverse sources, not big ones

Volume is a hardware problem. Diversity is a thinking problem, and it is the one this course sets. Two sources describing the same business can disagree in ways that look trivial and are not. They can hold different grains, one row per order against one row per line item, so a naive join multiplies your data instead of enriching it. They can refresh on different schedules, so a comparison quietly straddles two moments in time. They can use the same field name for different things, which is the most expensive disagreement of all because nothing looks wrong until the answer is.

The step almost everyone skips is writing down, in one sentence per source, what a single row represents. One row is one completed payment. One row is one customer as of the export date. One row is one product listing per region. Three sentences take five minutes and they determine every join, every aggregation and every count for the rest of the project. Where the sentences cannot be written, the source is not understood yet, and no amount of transformation will fix a misunderstanding at that level.

Matching is where diverse sources hurt most. When two systems share no key, you match on what they do share: names, dates, addresses, amounts. All of these are unreliable in specific, well known ways. Names carry spelling variants, initials, married names and inconsistent order. Addresses carry abbreviations and formatting. Dates carry timezone and locale ambiguity. A match rule needs a stated basis, a tie break for the cases where two candidates fit, and an honest count of records left unmatched, because the unmatched pile is data about your rule as much as about your data.

Shaping is the other half of the course description, and it has a direction. You do not shape data into what looks tidy, you shape it into what the consuming application requires. Ask what the downstream tool needs before you touch anything: one row per what, which fields mandatory, which types, what happens to a missing value when it arrives. That target is a contract, and wrangling is the work of honouring it.

Budgeting words when the pipeline is the argument

If your version of D497 is assessed by a performance assessment, expect some aspects to be demonstrated by the pipeline itself and others to be answered only in prose. WGU requires a score of 2 in each aspect and scores each independently, so a technically flawless pipeline with two unexplained decisions returns just as surely as a broken one.

Worked example, weighting by judgment. Take six scored aspects with a written component near 1,600 words alongside your code. Reserve 140 for a source inventory naming each source, its format, its grain and its refresh. That leaves 1,460 across six aspects, which flat is 243 each.

Flat is wrong here, because wrangling aspects are not equally interesting. Steps forced by the data, such as parsing a column that arrived as text, involve no choice and need only a sentence of justification. Steps where you chose between defensible options, such as how to handle missing values or how to match records without a key, are the whole point of the aspect. Fund four procedural aspects at 180 words each, which is 720, and two judgment aspects at 370 each, which is 740. Together that is 1,460 and the budget closes.

The rule to carry forward: budget by how much of the decision was yours. Where the data left you no option, say what you did and move on. Where you picked one reasonable approach over another reasonable approach, spend the words, because the alternative you rejected is the evidence that you understood the choice.

Shape for a wrangling deliverable

Where the work produces a pipeline plus documentation, these stages record what a reader needs to trust the shaped output. Where the directions name a format, the directions win.

Profiling comes before transformation in the table because it comes before transformation in reality. A cleaning rule written before anyone looked at the actual values is a guess, and guesses in this course tend to remove exactly the rows that made the dataset interesting.

Reproducibility is the evidence standard here

In a wrangling course the claim being made is that your output faithfully represents your input after known, deliberate changes. The only way to support that claim is a trail someone else could walk.

• Keep the raw extract untouched and read only. Every change happens in code that reads raw and writes somewhere else, so the starting point is always recoverable.
• Script every step, including the small ones. A correction made by hand in a spreadsheet is invisible, unrepeatable and impossible to audit, which makes it worthless as evidence however correct it was.
• Record counts entering and leaving each step, and explain any drop. Unexplained shrinkage is the fastest way to lose an evaluator's confidence in everything downstream.
• State locale, timezone and encoding assumptions explicitly, since these silently change values rather than throwing errors.
• Cite the documentation of the specific library or tool version whenever you assert behaviour, because parsing defaults and null handling differ between versions of the same package.
• Where data comes from outside the organization, record its provenance and the terms under which it may be used, in paraphrase with APA references, keeping quotation minimal because submissions run through a similarity check.

Missing values deserve a paragraph of their own because students routinely treat one problem as three identical blanks. A value that is absent because it was never collected, a value that is unknown to the person who supplied it, and a value that does not apply to that record are three different facts, and collapsing them into a single null destroys information you may need later. Where the source distinguishes them, preserve the distinction with a flag rather than a value. Where it does not, say so, and say what you assumed.

Imputation carries its own reporting duty. Filling a gap with an average keeps the row usable and makes the distribution narrower than reality, so anything computed from that column afterwards inherits an understatement of spread. Say which fields were imputed, by what method, and what proportion of each field it affected. A reader can accept a documented imputation. An undocumented one turns every later number into a question.

What clears a wrangling task

Competent work makes the pipeline legible. Sources are described by grain, not just by name. The target shape is stated before the transformations that produce it. Profiling precedes cleaning and the cleaning rules answer things the profiling actually found. Every step has a reason and a count. The matching rule states its basis and its tie break. Missing data policy is per field rather than global. Validation checks exist and their results are shown, including the ones that passed.

Returns follow a pattern. Rows disappear between stages with no accounting. A join on a shared name is presented without any statement of how duplicates were resolved. Dates are parsed with library defaults and quietly reinterpreted. Or the document describes what the code does, line by line, instead of why each decision was taken, which spends the entire word budget on material the evaluator can read in the script.

WGU records the outcome as Competent or Not Competent, with no letter grades and no ordinary grade point average, and the three competency units place this course inside a six month term charged at a flat rate, so finishing it earlier makes every other course in that term cheaper in effect. Performance assessment work can be revised and resubmitted with no grade penalty, so the right time to submit is when every aspect has a genuine answer. Where your course also carries an objective assessment, WGU objective assessments are proctored and our position is unchanged: we prepare only, with profiling drills, transformation practice and an honest read on your preassessment. We do not sit assessments, take no part during one, and never ask for or handle portal credentials.

Six mistakes that cost time in D497

• Editing the raw file. Once the original is modified there is no baseline to reconcile against, and the first time a rule needs revisiting the whole project restarts.
• Treating every blank as one thing. Not collected, unknown and not applicable are separate facts. Flag them separately while the distinction still exists in the source.
• Matching on names with no tie break. Two customers share a name eventually. A rule without a documented tie break silently merges two people into one record.
• Trusting date parsing defaults. An ambiguous day and month order is only detectable above the twelfth of the month, so a mis-parse can look correct for a third of your rows.
• Numbers arriving as text. Currency symbols, thousands separators and trailing spaces make a numeric column sort alphabetically, and the failure is silent rather than loud.
• Shaping for tidiness instead of for the consumer. The target shape is set by what the application needs. A structure that looks elegant and does not fit that contract has to be redone.

Support through the pipeline

Send the rubric, the task directions and your sources. Work starts with the grain sentences, one per source, because everything downstream depends on them and they take minutes. Then the target contract gets written down explicitly, then profiling output, then the transformation record with counts at every boundary. Only after that do we draft, and the draft is organised around decisions rather than around code. The walkthrough concentrates on the matching rule and the missing value policy, since those are the two places where an evaluator can see judgment or its absence most clearly.

D497 feeds directly into analysis, so it pairs well with D326 Advanced Data Management, and the scripting habits it rewards come from D493 Scripting and Programming Applications. If the rules about what may be done with the data are the part you are unsure of, that material sits in D494 Data and Information Governance.

Questions students ask about D497

Where D497 sits in WGU's programs

The July 2026 catalog places this code in 1 current WGU program. Open a program page for the complete standard path and term positions. The live Degree Plan remains authoritative after transfer credit, substitutions, and mentor planning.

The assessments, one by one

The public catalog does not publish this course's PA/OA identity or task count. WGU Tutors publishes at most one PA manual per course and only from a WGU-controlled public rubric. Until that source exists, PA help begins from the student's real Course of Study and OA support remains preparation only.

Keep going