🚀 Escaping the YouTube algorithm

I love watching YouTube videos that improve my life in some tangible way. Unfortunately, the YouTube algorithm doesn’t agree. It likes to feed me clickbait and other garbage.

This isn’t all that surprising. The algorithm prioritises clicks and watch time.

So I set out on a mission: Can I write code that will automatically find me valuable videos, eliminating my dependence on the YouTube algorithm?

Here’s how it went.

🗺️ The best laid plans

I started by visualising what I wanted the tool to do. …

What’s wrong with correlation?

In May 2016, the COMPAS algorithm was flagged as being racially biased [1]. This algorithm was used by the US to guide criminal sentencing by predicting likelihood of reoffending. It estimated a black person was more likely to re-offend than a white person with the same other background factors.

The problem was, the algorithm was mistaking correlation (patterns of crime in the past) with causation (that being black makes you more likely to commit a crime).

This can be a problem in medicine as well. Consider the following:

100 patients are admitted to hospital with pneumonia, of which 15 also…

Background

3 years ago, I had just finished medical school and started working full-time as a doctor in the UK’s National Health Service (NHS). Now, I work full-time as a data scientist at dunnhumby, writing code for “Big Data” analytics with Python and Spark.

More and more people are making the transition towards data science, or related technical roles, from a variety of disciplines. So in this article I’m going to share my experiences and advice for making a (perhaps) unconventional career transition into a technical role. I can break these down into five main learnings:

(1) Find technical friends

Coming from a medical background…

The cost of drug discovery

The drug development process is notoriously difficult; it takes on average 10–15 years to take a drug from identifying the molecule to being used clinically, and only approximately 3% of the initial drugs will make it all the way through. The cost of bringing a drug all the way through the development pipeline, only to fail at the final hurdle, typically exceed $1 billion.

For those not familiar with the drug development pathway, here’s a useful overview:

As the diagram shows, there are numerous stages where drugs can fail. It’s far better to fail early than fail late, so any…

Motivation

I’m currently looking for a new data science role, but have found it frustrating that there are so many different websites, which list different jobs and at different times. It was becoming laborious to continually check each website to see what new roles had been posted.

But then I remembered; I’m a data scientist. There must be an easier way to automate this process. So I decided to create a pipeline, which involved the following steps, and to automate part of the process using Python:

1. Extract all new job postings at a regular interval

I decided to write some Python code to web-scrape jobs from the websites I was…

Data Science in the Real World

We are seeing increasing use of, and discussion about, artificial intelligence (AI) and machine learning (ML) across all sectors; healthcare included. Greater volumes of data, improved computing power and break-throughs in machine learning techniques have provided fertile soils for innovation. In healthcare, this is reflected by an exponential increase in research involving AI, and a corresponding surge in publications and academic funding.

But do doctors really need to understand machine learning? We think so, and here are our top reasons why.

Communication with patients

We are beginning to see machine learning tools used in clinical settings and with our current wave of new…

I read an article a few months ago about how Jeff Bezos banned PowerPoints from meetings at Amazon. Instead, people write a “narratively structured six-page memo” to lay out what’s being discussed.

At the time of reading the article, I was creating a PowerPoint presentation for a group of medical students. I was delivering four sessions covering machine learning principles and their applications in healthcare. However, I was finding it challenging to condense complex ideas into bullet points on the slides. I was also struggling to consolidate my knowledge while doing so.

After reading the article I switched to the…

About 3 years ago, I developed a deep interest in learning to rap well. I emphasize the word ‘developed’ because up until this point, I was waiting to passively discover a passion which would somehow carry me towards expertise.

Authenticity

The main initial ingredient — from my experience — needed to produce expertise is authenticity. Authenticity is essential because any serious undertaking of a new skill will necessarily face you up against failure and self-doubt. The new skill needs to be important enough to make this tolerable. With rap, I reached a point where I needed to know my thoughts…