The Onion is Americas's Finest News Source -- a fantastic site of satirical news headlines. However, non-satirical news outlets seem to often compete with The Onion often blurring the lines on who is actually more absurd. Thus, a subreddit called r/NotTheOnion was born to collect headlines from supposedly real news stories that could be mistaken for Onion headlines. The subreddit is quite popular with about 22 million members and chalk full of headlines that really can be tough to distinguish from satire.
So, this is supposedly a difficult classification problem for humans which begs the question -- Can a machine learning model do better? 😉
import pandas as pd
import numpy as np
import re
import time
import requests
import json
import datetime
import tensorflow as tf
Data is extracted from Reddit using the Pushshift API as documented here: github.com/pushshift/api
The Pushshift API only returns at most 1000 posts with each request, so I create a list of dates to pull ~1000 posts between these dates.
dates_list = []
# Making list of dates; Each January 1st and June 1st from 2015 until January 1st 2020
for i in range(10,21):
dates_list.append('01/01/20'+str(i))
dates_list.append('01/06/20'+str(i))
# Popping June 2020 since it hasn't happened yet
dates_list.pop()
dates_list
Returns Unix timestamp that Pushshift API requires for dates
def getTimeStamp(date_input):
return time.mktime(datetime.datetime.strptime(date_input, "%d/%m/%Y").timetuple())
dates = [int(getTimeStamp(date)) for date in dates_list]
dates
Returns the top 1000 posts in the given subreddit between the given times.
Code modified from the following article:
medium.com/@RareLoot/using-pushshifts-api-to-extract-reddit-submissions-fb517b286563
def getPushshiftData(after, before, sub):
url = ('https://api.pushshift.io/reddit/search/submission/?size=1000&after='+
str(after)+'&before='+str(before)+'&subreddit='+str(sub)+'&sort_type=score'+'&sort=desc')
print(url)
r = requests.get(url)
data = json.loads(r.text)
return data['data']
Here I loop through all the dates above and get the top ~1000 posts in the chosen subreddit.
I end up with 9065 Onion headlines and 15432 "fake"-Onion headlines from r/NotThenOnion.
I then keep the first 9000 and first 15000 for easier batching.
def getTitles(subreddit):
titles_new = []
titles = []
for i in range(len(dates)-1):
# Setting up dates
after = dates[i]
before = dates[i+1]
# Getting subreddit data between the dates after and before from r/NotTheOnion
raw_json = getPushshiftData(after,before,subreddit)
# Extracting just the title
titles_new = [post['title'] for post in raw_json]
# Appending new data on
titles = titles + titles_new
# A few posts were extracted twice, set gets rid of duplicates
titles = list(set(titles))
return titles
not_onion = getTitles('nottheonion')
onion = getTitles('theonion')
onion = onion[:9000]
not_onion = not_onion[:15000]
Labeling Onion headlines as 1, and r/NotTheOnion headlines as 0.
df1= pd.DataFrame({'text':onion})
df1['label'] = 1
df2 = pd.DataFrame({'text':not_onion})
df2['label'] = 0
# Combining both datasets
df = pd.concat([df1,df2])
# Shuffling the dataset
df = df.sample(frac=1).reset_index(drop=True)
# Converting all text to lowercase, fixing ampersands and getting rid
# of dashes and apostrophes as they can mess up the dictionary
df['text'] = df['text'].str.lower()
df['text'] = df['text'].str.replace(r'&', 'and')
df['text'] = df['text'].str.replace(r'-', ' ')
df['text'] = df['text'].str.replace(r'[^\s\w]','')
# Saving the dataframe to a csv file
df.to_csv('OnionOrNot.csv')
df.head()
| text | label | |
|---|---|---|
| 0 | egypt funeral turns happy after dead man awakes | 0 |
| 1 | michael jordan attacks softness lack of compet... | 1 |
| 2 | feds bust massive child pornography corporatio... | 1 |
| 3 | joker arrested for flashing his penis but it... | 0 |
| 4 | mexican lawmaker wants to fine people 1600 for... | 0 |
Running this when I return to the project so I don't have to use the Pushshift API etc. again.
df = pd.read_csv('OnionOrNot.csv', index_col = 0)
df.head()
| text | label | |
|---|---|---|
| 0 | egypt funeral turns happy after dead man awakes | 0 |
| 1 | michael jordan attacks softness lack of compet... | 1 |
| 2 | feds bust massive child pornography corporatio... | 1 |
| 3 | joker arrested for flashing his penis but it... | 0 |
| 4 | mexican lawmaker wants to fine people 1600 for... | 0 |
vocab_set = set()
sentence_lengths = []
for i in range(len(df)):
# Updates adds all items to the set, re.split splits the text into words
sentence_words = re.split(r'\s',df.iloc[i]['text'])
vocab_set.update(sentence_words)
sentence_lengths.append(len(sentence_words))
This way we can map the words in the dataframe to lists of numbers
vocab_list = list(vocab_set)
vocab_dict = {vocab_list[i-1]: i for i in range(1, len(vocab_list)+1)}
max_length = max(sentence_lengths)
def toNumbers(row):
words = re.findall(r'([\w]+)', row['text'])
nums = np.array([vocab_dict[words[j]] for j in range(len(words))])
return np.pad(nums, (0, max_length - len(nums)), mode='constant')
nums = df.apply(lambda row: toNumbers(row), axis=1)
df['nums'] = nums
df['nums'].head()
labels = np.asarray(df['label'].values)
features = np.stack(df['nums'].values)
features.shape, labels.shape
def get_compiled_model():
embedding_dim=16
model = tf.keras.Sequential([
tf.keras.layers.Embedding(len(vocab_set)+1, 64),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1, activation='sigmoid')
])
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
return model
model = get_compiled_model()
model.fit(features, labels, batch_size=32, epochs=5, verbose=2, validation_split=0.2);
embedding_dim=16
model = keras.Sequential([
layers.Embedding(len(vocab_set)+1, embedding_dim),
layers.GlobalAveragePooling1D(),
layers.Dense(16, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
No discernable change in accuracy tuning embedding_dim
embedding_dim=32
model = keras.Sequential([
layers.Embedding(len(vocab_set)+1, embedding_dim),
layers.GlobalAveragePooling1D(),
layers.Dense(16, activation='relu'),
layers.Dense(1, activation='sigmoid')
])
Starts overfitting after epoch 1
embedding_dim=16
model = tf.keras.Sequential([
tf.keras.layers.Embedding(len(vocab_set)+1, 64),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(1, activation='sigmoid')
])
Again starts overfitting after epoch 1
embedding_dim=16
model = tf.keras.Sequential([
tf.keras.layers.Embedding(len(vocab_set)+1, 64),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32)),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(1, activation='sigmoid')
])
So, playing around with some simple models and parameters it appears we can get about 87% validation accuracy. I haven't done a true deep dive but after shuffling the "real" vs "fake" headlines and investigating myself I seem to be about that accurate, perhaps a bit more!
You can download the CSV of all the data in my GitHub repo for the project (or of course follow along above to get the data yourself!). If you check it out let me know if you try any models and beat my "benchmarks" above 😜. You can always reach me by the email in the footer!