Course Description
CAP4786 is the big data analytics course. The Statewide Course Numbering System titles it Big Data Analytics and defines it tightly: "this course introduces students to the handling of big data on Hadoop's MapReduce environment. Advanced data mining/machine learning applications created using Spark." The statewide prerequisites are a database course and a data structures or programming course, with minimum grade conditions.
⚠ That definition names specific technologies, which is unusual and worth noticing. Most statewide descriptions describe a subject; this one describes a stack — Hadoop, MapReduce, Spark. ⚠⚠ And that dates it: MapReduce as a programming model has been largely superseded in practice by Spark and by cloud-native engines, and a course that spent its weeks writing MapReduce jobs would be teaching something students will rarely meet. Expect your own course to have moved on, and read its description rather than the statewide one.
Two Florida public universities carry it, both at 3 credits:
| Institution | Its title | Credits |
|---|
| Florida Polytechnic University | Topics in Big Data Analytics | 3 |
| University of West Florida | Introduction to Big Data Analytics | 3 |
⚠ "Topics in" is a meaningful phrase — see the offering notes, because it changes what the transcript line tells a reader.
The underlying idea is simple and worth stating: big data is not a size, it is a threshold. Data becomes "big" when it no longer fits the tools you would otherwise use — when it will not sit in memory, or on one machine, or be processed in acceptable time by one processor. ⚠ Everything in this course follows from that: distribution, parallelism, fault tolerance and the design compromises each forces.
Learning Outcomes
Required Outcomes
- Explain what makes a problem a "big data" problem — volume, velocity, variety — and recognise when a conventional database is the better answer.
- Explain distributed storage: how a file system such as HDFS partitions and replicates data across nodes, and why replication is a correctness requirement rather than a convenience.
- Explain the MapReduce programming model — map, shuffle, reduce — and express a problem in it.
- Explain why moving computation to data matters when the data will not move.
- Use Apache Spark: RDDs and DataFrames, transformations and actions, and the consequence of lazy evaluation.
- Explain Spark's execution model — driver, executors, partitions, stages — and read the execution plan of a job.
- Identify and address the characteristic performance problems: data skew, expensive shuffles, and unnecessary materialisation.
- Perform distributed data preparation: loading, cleaning, joining and aggregating at scale.
- Apply machine learning on a distributed platform using Spark MLlib — classification, regression, clustering — and evaluate the result honestly.
- Explain how a distributed system handles node failure, and what that costs.
- Compare batch and streaming processing and select between them for a stated requirement.
- Evaluate whether a distributed solution is warranted at all — ⚠ including the cost, operational and complexity penalties of choosing one when it is not.
Optional Outcomes
- Work with NoSQL stores — key-value, document, column-family, graph — and their consistency models.
- Build a streaming pipeline with Kafka and Spark Structured Streaming.
- Work on a cloud platform — EMR, Databricks, Dataproc — and reason about its cost model.
- Address the CAP theorem and eventual consistency.
- Apply graph processing or recommendation algorithms at scale.
- Address data governance, privacy and regulatory constraints on large personal data sets.
- Complete an end-to-end project on a genuinely large public data set.
Major Topics
Required Topics
- Foundations — what "big data" means operationally; the limits of single-machine processing; scale-up against scale-out.
- Distributed storage — HDFS architecture, blocks, replication, the name node, and object storage as its cloud successor.
- MapReduce — the model, the shuffle, combiners and partitioners; expressing joins and aggregations in it.
- The Hadoop ecosystem — YARN, Hive, HBase and where each fits.
- Apache Spark — architecture, RDDs and DataFrames, lazy evaluation, caching, the DAG scheduler.
- Spark SQL — structured processing and the Catalyst optimiser.
- Performance — partitioning, shuffle cost, broadcast joins, data skew and its mitigation, serialisation formats (Parquet, ORC).
- Distributed machine learning — MLlib pipelines, feature engineering at scale, training and evaluation, and where distribution helps and where it does not.
- Fault tolerance — lineage, recomputation, speculative execution.
- Batch and streaming — the distinction, micro-batching, windowing, late data.
- Judgement — ⚠ when the right answer is a single machine, a database, or a laptop with pandas.
Optional Topics
- NoSQL data stores and consistency models.
- Kafka and streaming pipelines.
- Cloud platforms and cost management.
- CAP theorem and distributed systems theory.
- Graph processing and recommenders.
- Privacy, governance and regulated data at scale.
- An end-to-end capstone project.
Resources & Tools
- Learning Spark (Damji et al., O'Reilly) is the current standard and matches modern practice; Spark: The Definitive Guide is the fuller reference.
- Hadoop: The Definitive Guide by Tom White remains the reference for the Hadoop layer — ⚠ useful for understanding HDFS and MapReduce as history and foundation rather than as current practice.
- Mining of Massive Datasets (Leskovec, Rajaraman, Ullman) is free online and is the best treatment of the algorithms behind the platforms.
- ⚠ Everything needed is free. Spark runs locally in standalone mode on a laptop, which is enough to learn the programming model; Databricks Community Edition gives a free hosted cluster; and the major cloud providers offer student credits. Do not assume you need a cluster to start.
- Languages: Python (PySpark) dominates teaching and increasingly industry; Scala is Spark's native language and appears in performance-critical work.
- ⚠ Genuinely large public data sets make the difference between a real project and a toy one: the NYC taxi trip records, Common Crawl, GDELT, and the US Census microdata are all free and are actually large enough to require distribution.
- ⚠ Watch cloud costs. A misconfigured cluster left running is the classic student mistake and can produce a real bill. Set budget alerts before launching anything.
Career Pathways
- Data Engineer — ⚠ the direct destination and the strongest job market in this area; pipelines and distributed processing are the daily work.
- Data Scientist (SOC 15-2051) — where the analysis rather than the infrastructure is the focus.
- Database Architect (SOC 15-1243) and Software Developer (SOC 15-1252) in data platform roles.
- Machine Learning Engineer — with further study.
- Florida employers: healthcare systems, which hold very large clinical data sets under regulatory constraint; Publix and the large retail and logistics operations; financial services in South Florida; the theme park and hospitality groups, which run substantial personalisation and operations analytics; the defence and simulation cluster in Central Florida; and NASA and the space sector, where telemetry and Earth-observation data are genuinely large.
- ⚠ An honest note on scale: most organisations do not have big data in the technical sense, and many "big data" roles are in practice data engineering on datasets a good server could hold. That is not a criticism of the course — the distributed-systems reasoning transfers — but a graduate should be able to say when distribution is unnecessary, which is a more valuable judgement than being able to configure a cluster.
Special Information
Offering Notes — offerings and hours, school by school
| Institution | Its title | Credits | Contact hours |
|---|
| Florida Polytechnic University | Topics in Big Data Analytics | 3 | not published |
| University of West Florida | Introduction to Big Data Analytics | 3 | not published |
Both are State University System institutions, so statewide numbering guarantees transfer between them. ✅ Both carry it at 3 credits.
⚠ The 45 contact hours at the top of this guide are derived — the Florida convention for a 3-credit lecture course. Neither institution publishes an hour figure.
⚠⚠ "Topics in" signals variable content — and that follows you onto the transcript
Florida Polytechnic titles it "Topics in Big Data Analytics." ⚠ "Topics in" conventionally means the content varies by term and by instructor, within the subject area.
Two consequences a student should know.
- What you study depends on when you take it. One term may emphasise streaming, another distributed machine learning, another platform engineering. Read the term's own syllabus; this guide's topic list is the statewide subject, not a promise about your section.
- ⚠ The transcript line conveys nothing about what you covered. A receiving institution or a graduate programme evaluating "Topics in Big Data Analytics" cannot tell what was taught, and will ask. Keep the syllabus and the reading list — they are the only record.
⚠ A variable-content course is sometimes also repeatable for credit under different topics. If that applies at your institution, the degree audit rules matter — check before enrolling twice.
⚠⚠ The statewide definition names a technology stack, and the stack has moved
The state describes "Hadoop's MapReduce environment" with Spark for the advanced work. ⚠ That was an accurate description of the field around 2015 and is no longer where practice sits. MapReduce is rarely written directly now; Spark superseded it for most workloads, and cloud-native engines and lakehouse platforms have taken much of the rest.
What that means for a student:
- MapReduce is still worth understanding — as the model that established how distributed processing works, and because Spark's shuffle behaviour only makes sense once you know it. ⚠ But a course spending most of its weeks writing MapReduce jobs is preparing you for an interview question rather than a job.
- Spark is the employable skill, and PySpark specifically.
- Cloud platform experience is what postings ask for — Databricks, EMR, Glue, BigQuery. If your course does not include it, the free tiers make it a weekend's work to add.
⚠ This is a good example of why a statewide definition should describe a subject rather than a product. A definition naming technologies dates predictably; one naming "distributed processing of data at scale" would not have.
Prerequisites, and the one that matters
The statewide prerequisites are a database course (COP4710) and a data structures or programming course, both with minimum grade conditions. ⚠ The data structures requirement is the load-bearing one: distributed processing is applied data-structures reasoning — partitioning, hashing, sorting and the cost of moving data — and a student without it will be typing commands rather than understanding them. Programming fluency is assumed, not taught.
Position in the curriculum and workload
A 4000-level course late in a computing degree, pairing naturally with data warehousing and data visualisation.
Budget ten to fourteen hours a week. ⚠ Environment setup is a real and underestimated cost — getting Spark, Java, Python and their versions to agree is a rite of passage that has consumed many first weeks. Use a managed environment such as Databricks Community Edition if the course permits it, and spend the time on the concepts instead.
AI Integration
⚠ There is a neat reflexivity here: the distributed training of large models is itself one of the largest big-data workloads in existence, so this course teaches the infrastructure that produces the tools discussed in this section.
Genuinely useful: writing and explaining PySpark and SQL, which is legitimate professional practice; explaining an execution plan or a stage breakdown, which is genuinely hard to read at first; diagnosing an error message from a distributed job, where the useful line is buried in hundreds; explaining a concept a second way — lazy evaluation and the shuffle are the standard sticking points; and drafting documentation.
⚠⚠ Where it fails:
- Code that works on a small sample and fails at scale. ⚠ This is the characteristic failure of this subject. A generated job may collect to the driver, trigger an enormous shuffle, or produce catastrophic skew — none of which is visible on test data, all of which surfaces on a cluster and costs money.
- Version and API confusion. Spark's APIs have changed substantially across versions; RDD-era code and current DataFrame code are mixed freely in generated answers.
- Recommending distribution when it is unnecessary. Ask how to process a dataset and a model will reach for Spark, when a single machine with pandas or DuckDB would be faster, simpler and free. ⚠ Knowing you do not need a cluster is a senior judgement, and the tools do not have it.
- No knowledge of your cluster, your data's distribution or your budget — the three things that determine whether a job is a good idea.
⚠ The professional habit: read the execution plan and test on a representative sample before running on the full dataset. That is standard practice for cost reasons alone, it applies to your own code as much as to generated code, and it catches almost everything above.
Academic integrity: read your syllabus; policies on AI-assisted code vary sharply between instructors in computing, and where a project is defended, work a student cannot explain is exposed.