diff --git a/02_activities/assignments/DC_Cohort/Assignment1.md b/02_activities/assignments/DC_Cohort/Assignment1.md index f650c9752..5fd3a0df1 100644 --- a/02_activities/assignments/DC_Cohort/Assignment1.md +++ b/02_activities/assignments/DC_Cohort/Assignment1.md @@ -209,5 +209,17 @@ Consider, for example, concepts of fariness, inequality, social structures, marg ``` -Your thoughts... +One data system I encounter in my day-to-day life is a medical record. These records include patient information such as name, sex, marital status, work +place, place of residence, health card number, and emergency contacts (these would typically be family members). They also include medical history +of a patient such as past immunizations, surgeries, doctor visits, lab test results, and referrals. On top of that, they record family history of +illnesses and mental health conditions. If a doctor is unsure of why a patient has certain symptoms (sometimes the lab results and subjective description +of patient's symptoms do not match), the doctor may visit the family history data and test the patient for those conditions that run in the family that +closely resemble the symptoms. It is important for doctors to have access to family records of a patient to identify risks and test for them. However, +those with weak or without family ties such as orphans, would have incomplete family medical records, making it harder to identify risk factors and +diagnose. Another interesting aspect is culture. In many cultures, certain conditions (especially related to mental health) are considered to be +embarrassing or "not real"; for this reason, they wouldn't report this information to their doctor and the family history data would be incomplete. +On the flip side, the doctors may also rely too much on family history information and dismiss patients' thoughts about their own condition, which +may especially be true for younger patients and recent immigrants who may struggle communicating their thoughts due to language barrier. This way, +medical databases and social structures are intertwined. + ``` diff --git a/02_activities/assignments/DC_Cohort/Assignment2.md b/02_activities/assignments/DC_Cohort/Assignment2.md index 01f991d02..9d5fd1438 100644 --- a/02_activities/assignments/DC_Cohort/Assignment2.md +++ b/02_activities/assignments/DC_Cohort/Assignment2.md @@ -56,7 +56,8 @@ The store wants to keep customer addresses. Propose two architectures for the CU **HINT:** search type 1 vs type 2 slowly changing dimensions. ``` -Your answer... +Type 2 would be the architecture that retains the changes, meanwhile type 1 overwrites old data with new data. +For the type 2, CUSTOMER_ADDRESS table could contain attributes such as address, start_date, and end_date. Since there are start and end dates of using a given address, we could keep adding a new address without losing information about the old address. For the type 1, CUSTOMER_ADDRESS table would not have these dates, so each time that a customer changes their address, this information is overwritten. ``` *** @@ -187,9 +188,11 @@ Read: Boykis, V. (2019, October 16). _Neural nets are just people all the way do **What are some of the ethical issues important to this story?** -Consider, for example, concepts of labour, bias, LLM proliferation, moderating content, intersection of technology and society, ect. +Consider, for example, concepts of labour, bias, LLM proliferation, moderating content, intersection of technology and society, etc. ``` -Your thoughts... +I found it interesting how it requires heavy human labor such as image labeling, to train the machine to recognize items. Many think that machines make our lives easy and are even capable of replacing people in some sectors. However, there is a lot of manual labor involved to get to that stage. Since it requires so much work, the manual workers are often recruited from underdeveloped countries such as Kenya and Venezuela. These are educated but unemployed individuals that are actively seeking a job to sustain themselves and their families. However, there are reports of these people getting taken advantage of by the big tech companies in America; they are underpaid and overworked. From my own experience, I know how physically hard it is to do manual scoring. For my project, I have to count neurons in different brain regions. Since there are many cells, I was initially planning to rely on machine learning tools; all I needed to do was to train the machine how a cell looks like using images of real cells. However, even after the training, the machine still couldn't reliably identify all cells (for example, it would count a block of clustered cells as one cell). Now, I have to manually count every cell in each image (there can be anywhere between 1500-2000 cells in an image), which is very hard on my eyes sometimes. + +There is also a lot of subjectivity involved in manual labeling. As it was stated in the article, some of the labels are inappropriate and wrong. If someone happens to be biased against certain groups of people or has a certain worldview, which is very likely considering there is a certain group of people that are more likely to get hired than others, they would label images accordingly. Since the machine is not perfect, these labels may be exarcebated further, which raises ethical concerns. ``` diff --git a/02_activities/assignments/DC_Cohort/Farmers_market_log_model.pdf b/02_activities/assignments/DC_Cohort/Farmers_market_log_model.pdf new file mode 100644 index 000000000..cddb0cdce Binary files /dev/null and b/02_activities/assignments/DC_Cohort/Farmers_market_log_model.pdf differ diff --git a/02_activities/assignments/DC_Cohort/assignment1.sql b/02_activities/assignments/DC_Cohort/assignment1.sql index 2ec561e2a..32676bb4f 100644 --- a/02_activities/assignments/DC_Cohort/assignment1.sql +++ b/02_activities/assignments/DC_Cohort/assignment1.sql @@ -6,9 +6,7 @@ --SELECT /* 1. Write a query that returns everything in the customer table. */ --QUERY 1 - - - +SELECT * FROM customer --END QUERY @@ -16,9 +14,10 @@ /* 2. Write a query that displays all of the columns and 10 rows from the customer table, sorted by customer_last_name, then customer_first_ name. */ --QUERY 2 - - - +SELECT customer_id, customer_first_name, customer_last_name, customer_postal_code +FROM customer +ORDER BY customer_last_name, customer_first_name +LIMIT 10 --END QUERY @@ -27,9 +26,10 @@ sorted by customer_last_name, then customer_first_ name. */ /* 1. Write a query that returns all customer purchases of product IDs 4 and 9. Limit to 25 rows of output. */ --QUERY 3 - - - +SELECT * +FROM customer_purchases +WHERE product_id IN (4,9) +LIMIT 25 --END QUERY @@ -42,9 +42,11 @@ filtered by customer IDs between 8 and 10 (inclusive) using either: Limit to 25 rows of output. */ --QUERY 4 - - - +SELECT * +,quantity * cost_to_customer_per_qty AS price +FROM customer_purchases +WHERE customer_id BETWEEN 8 AND 10 +LIMIT 25 --END QUERY @@ -55,8 +57,10 @@ Using the product table, write a query that outputs the product_id and product_n columns and add a column called prod_qty_type_condensed that displays the word “unit” if the product_qty_type is “unit,” and otherwise displays the word “bulk.” */ --QUERY 5 - - +SELECT product_id, product_name, + CASE WHEN product_qty_type = "unit" THEN "unit" + ELSE "bulk" + END as prod_qty_type_condensed, --END QUERY @@ -66,9 +70,10 @@ if the product_qty_type is “unit,” and otherwise displays the word “bulk. add a column to the previous query called pepper_flag that outputs a 1 if the product_name contains the word “pepper” (regardless of capitalization), and otherwise outputs 0. */ --QUERY 6 - - - + CASE WHEN product_name like '%pepper%' THEN 1 + ELSE 0 + END as pepper_flag +FROM product --END QUERY @@ -78,9 +83,12 @@ contains the word “pepper” (regardless of capitalization), and otherwise out vendor_id field they both have in common, and sorts the result by market_date, then vendor_name. Limit to 24 rows of output. */ --QUERY 7 - - - +SELECT * +FROM vendor +INNER JOIN vendor_booth_assignments + ON vendor.vendor_id = vendor_booth_assignments.vendor_id +ORDER BY market_date, vendor_name +LIMIT 24 --END QUERY @@ -92,9 +100,10 @@ Limit to 24 rows of output. */ /* 1. Write a query that determines how many times each vendor has rented a booth at the farmer’s market by counting the vendor booth assignments per vendor_id. */ --QUERY 8 - - - +SELECT vendor_id, +COUNT(booth_number) +FROM vendor_booth_assignments +GROUP BY vendor_id --END QUERY @@ -105,9 +114,13 @@ of customers for them to give stickers to, sorted by last name, then first name. HINT: This query requires you to join two tables, use an aggregate function, and use the HAVING keyword. */ --QUERY 9 - - - +SELECT customer.customer_last_name, customer.customer_first_name +FROM customer +INNER JOIN customer_purchases + ON customer.customer_id = customer_purchases.customer_id +GROUP BY customer.customer_id, customer.customer_last_name, customer.customer_first_name +HAVING SUM(customer_purchases.cost_to_customer_per_qty) > 2000 +ORDER BY customer.customer_last_name, customer.customer_first_name --END QUERY @@ -124,36 +137,28 @@ When inserting the new vendor, you need to appropriately align the columns to be VALUES(col1,col2,col3,col4,col5) */ --QUERY 10 - - - - ---END QUERY - - --- Date -/*1. Get the customer_id, month, and year (in separate columns) of every purchase in the customer_purchases table. - -HINT: you might need to search for strfrtime modifers sqlite on the web to know what the modifers for month -and year are! -Limit to 25 rows of output. */ ---QUERY 11 - - - +DROP TABLE IF EXISTS temp.new_vendor; +CREATE TABLE temp.new_vendor AS +SELECT * +FROM vendor; +INSERT INTO temp.new_vendor(vendor_id, vendor_name, vendor_type, vendor_owner_first_name, vendor_owner_last_name) +VALUES(10, "Thomas Superfood Store", "Fresh Focused store", "Thomas", "Rosenthal") --END QUERY +/* SECTION 4 */ +/* What values systems are embedded in databases and data systems you encounter in your day-to-day life?*/ + +/*One data system I encounter in my day-to-day life is a medical record. These records include patient information such as name, sex, marital status, work +place, place of residence, health card number, and emergency contacts (these would typically be family members). They also include medical history +of a patient such as past immunizations, surgeries, doctor visits, lab test results, and referrals. On top of that, they record family history of +illnesses and mental health conditions. If a doctor is unsure of why a patient has certain symptoms (sometimes the lab results and subjective description +of patient's symptoms do not match), the doctor may visit the family history data and test the patient for those conditions that run in the family that +closely resemble the symptoms. It is important for doctors to have access to family records of a patient to identify risks and test for them. However, +those with weak or without family ties such as orphans, would have incomplete family medical records, making it harder to identify risk factors and +diagnose. Another interesting aspect is culture. In many cultures, certain conditions (especially related to mental health) are considered to be +embarrassing or "not real"; for this reason, they wouldn't report this information to their doctor and the family history data would be incomplete. +On the flip side, the doctors may also rely too much on family history information and dismiss patients' thoughts about their own condition, which +may especially be true for younger patients and recent immigrants who may struggle communicating their thoughts due to language barrier. This way, +medical databases and social structures are intertwined.*/ -/* 2. Using the previous query as a base, determine how much money each customer spent in April 2022. -Remember that money spent is quantity*cost_to_customer_per_qty. - -HINTS: you will need to AGGREGATE, GROUP BY, and filter... -but remember, STRFTIME returns a STRING for your WHERE statement... -AND be sure you remove the LIMIT from the previous query before aggregating!! */ ---QUERY 12 - - - - ---END QUERY diff --git a/02_activities/assignments/DC_Cohort/assignment2.sql b/02_activities/assignments/DC_Cohort/assignment2.sql index f7515f625..4509b26f4 100644 --- a/02_activities/assignments/DC_Cohort/assignment2.sql +++ b/02_activities/assignments/DC_Cohort/assignment2.sql @@ -22,8 +22,10 @@ The `||` values concatenate the columns into strings. Edit the appropriate columns -- you're making two edits -- and the NULL rows will be fixed. All the other rows will remain the same. */ --QUERY 1 - - +SELECT +product_name || ', ' || +COALESCE(product_size, '') || ' (' || COALESCE(product_qty_type, 'unit') || ')' +FROM product; --END QUERY @@ -40,10 +42,10 @@ each new market date for each customer, or select only the unique market dates p HINT: One of these approaches uses ROW_NUMBER() and one uses DENSE_RANK(). Filter the visits to dates before April 29, 2022. */ --QUERY 2 - - - - +SELECT customer_id, market_date, +DENSE_RANK() OVER(PARTITION BY customer_id ORDER BY market_date) AS visits +FROM customer_purchases +WHERE market_date < "2022-04-29" --END QUERY @@ -52,9 +54,11 @@ then write another query that uses this one as a subquery (or temp table) and fi only the customer’s most recent visit. HINT: Do not use the previous visit dates filter. */ --QUERY 3 - - - +SELECT customer_id, market_date +FROM (SELECT customer_id, market_date, + DENSE_RANK() OVER(PARTITION BY customer_id ORDER BY market_date DESC) AS recent_visits + FROM customer_purchases) AS visit_table +WHERE recent_visits = 1 --END QUERY @@ -65,9 +69,10 @@ customer_purchases table that indicates how many different times that customer h You can make this a running count by including an ORDER BY within the PARTITION BY if desired. Filter the visits to dates before April 29, 2022. */ --QUERY 4 - - - +SELECT customer_id, product_id, transaction_time, + COUNT(*) OVER(PARTITION BY customer_id, product_id ORDER BY transaction_time) AS purchases_values +FROM customer_purchases +WHERE market_date < "2022-04-29" --END QUERY @@ -84,18 +89,22 @@ Remove any trailing or leading whitespaces. Don't just use a case statement for Hint: you might need to use INSTR(product_name,'-') to find the hyphens. INSTR will help split the column. */ --QUERY 5 - - - +SELECT DISTINCT +product_name, + CASE WHEN INSTR(product_name, '-') > 0 + THEN TRIM(SUBSTR(product_name, INSTR(product_name, '-') + 1)) + ELSE NULL + END AS new_description +FROM product --END QUERY /* 2. Filter the query to show any product_size value that contain a number with REGEXP. */ --QUERY 6 - - - +SELECT * +FROM product +WHERE product_size REGEXP '[0-9]' --END QUERY @@ -110,14 +119,26 @@ HINT: There are a possibly a few ways to do this query, but if you're struggling 3) Query the second temp table twice, once for the best day, once for the worst day, with a UNION binding them. */ --QUERY 7 - - - + WITH sales AS + (SELECT market_date, SUM(cost_to_customer_per_qty) AS total_sales + FROM customer_purchases + GROUP BY market_date), +sale_rank AS + (SELECT *, + ROW_NUMBER() OVER (ORDER BY total_sales ASC) AS lowest_sales, + ROW_NUMBER() OVER (ORDER BY total_sales DESC) AS highest_sales + FROM sales) +SELECT market_date, total_sales +FROM sale_rank +WHERE lowest_sales = 1 +UNION +SELECT market_date, total_sales +FROM sale_rank +WHERE highest_sales = 1 --END QUERY - /* SECTION 3 */ -- Cross Join @@ -132,8 +153,18 @@ How many customers are there (y). Before your final group by you should have the product of those two queries (x*y). */ --QUERY 8 - - +SELECT + vendor.vendor_name, + product.product_name, + SUM(5 * vendor_inventory.original_price) AS money_made +FROM vendor_inventory +JOIN product + ON vendor_inventory.product_id = product.product_id +JOIN vendor + ON vendor_inventory.vendor_id = vendor.vendor_id +CROSS JOIN customer +GROUP BY vendor.vendor_name, product.product_name +ORDER BY vendor.vendor_name, product.product_name --END QUERY @@ -144,9 +175,12 @@ This table will contain only products where the `product_qty_type = 'unit'`. It should use all of the columns from the product table, as well as a new column for the `CURRENT_TIMESTAMP`. Name the timestamp column `snapshot_timestamp`. */ --QUERY 9 - - - +DROP TABLE IF EXISTS temp.product_units; +CREATE TEMP TABLE product_units AS + SELECT *, + CURRENT_TIMESTAMP AS snapshot_timestamp +FROM product +WHERE product_qty_type = 'unit' --END QUERY @@ -154,21 +188,19 @@ Name the timestamp column `snapshot_timestamp`. */ /*2. Using `INSERT`, add a new row to the product_units table (with an updated timestamp). This can be any product you desire (e.g. add another record for Apple Pie). */ --QUERY 10 - - - +INSERT INTO product_units +(product_id, product_name, product_size, product_category_id, product_qty_type, snapshot_timestamp) +VALUES(1, 'Eggs', '2 dozen', 6, 'unit', CURRENT_TIMESTAMP) --END QUERY -- DELETE /* 1. Delete the older record for the whatever product you added. - HINT: If you don't specify a WHERE clause, you are going to have a bad time.*/ --QUERY 11 - - - +DELETE FROM product_units +WHERE product_name = 'Eggs' AND snapshot_timestamp = '2026-04-07 17:51:44' --END QUERY @@ -190,9 +222,13 @@ Finally, make sure you have a WHERE statement to update the right row, you'll need to use product_units.product_id to refer to the correct row within the product_units table. When you have all of these components, you can run the update statement. */ --QUERY 12 - - - +ALTER TABLE product_units +ADD current_quantity INT; +UPDATE product_units +SET current_quantity = + (SELECT COALESCE(MAX(vendor_inventory.quantity), 0) + FROM vendor_inventory + WHERE vendor_inventory.product_id = product_units.product_id); --END QUERY diff --git a/02_activities/assignments/DC_Cohort/log_model_bookstore.pdf b/02_activities/assignments/DC_Cohort/log_model_bookstore.pdf new file mode 100644 index 000000000..4aa5eecca Binary files /dev/null and b/02_activities/assignments/DC_Cohort/log_model_bookstore.pdf differ