Art & Intelligence
Human Sight¶
Over
99%of humanity can see one million different shades of color.
1%of us are gifted with a fourth photoreceptor cone, and can see 100 million colors.
We have an amazing ability to distinguish between subtle color tones.
But the human brain evolved to simplify the chaotic world around us, and to categorize and group what we see - including colors.
Rather than describing the sky above us as the tapestry of shades that it is, we instead say that the sky is blue.
Likewise with autumn leaves, which take on thousands and thousands of colors – we instead say brown, orange, and yellow.
The human experience of color is, in fact, a clustering algorithm.¶
And teaching a machine to emulate this approach can be accomplished in many ways.
And one of these is three-dimensional K-Means clustering.
Let's teach a machine to do exactly that.¶
import binascii
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from PIL import Image
import scipy.cluster
import scipy.stats as ss
First...¶
We'll extract every pixel from an image, and access each one's RGB values.
im = Image.open('image.jpg')
im
Next...¶
We'll cast the RGB values into an array, then resize and reshape.
im = im.resize((300, 300))
ar = np.asarray(im)
shape = ar.shape
ar = ar.reshape(np.prod(shape[:2]), shape[2]).astype(float)
Now...¶
We're ready to cluster. This time we'll choose
K=4clusters.
codes, dist = scipy.cluster.vq.kmeans(ar, 4)
We'll assign¶
each 3D-mapped pixel to the closest cluster center...
vectors, distance = scipy.cluster.vq.vq(ar, codes)
And we'll count...¶
how many pixels fell into each cluster.
counts, bins = np.histogram(vectors, len(codes))
Lastly...¶
Let's organize all this data into a pandas dataframe.
colors = dict(zip(ss.rankdata(-counts), codes.tolist()))
colors = {int(k): {'rgb': v} for k, v in colors.items()}
for i, v in enumerate(colors):
colors[v]['count'] = counts[i]
for v in colors.values():
v['rgb'] = [round(n) for n in v['rgb']]
v['hex'] = f"#{binascii.hexlify(bytearray(int(c) for c in v['rgb'])).decode('ascii')}"
v['r'] = v['rgb'][0]
v['g'] = v['rgb'][1]
v['b'] = v['rgb'][2]
df = pd.DataFrame.from_dict(
data=colors, orient='index').reset_index().rename(
columns={'index': 'rank'}).sort_values(by='rank')
df
| rank | rgb | count | hex | r | g | b | |
|---|---|---|---|---|---|---|---|
| 2 | 1 | [5, 106, 163] | 52719 | #056aa3 | 5 | 106 | 163 |
| 1 | 2 | [13, 22, 32] | 15268 | #0d1620 | 13 | 22 | 32 |
| 3 | 3 | [93, 152, 191] | 14049 | #5d98bf | 93 | 152 | 191 |
| 0 | 4 | [247, 180, 91] | 7964 | #f7b45b | 247 | 180 | 91 |
Huzzah!¶
We've just discovered the dominant colors in an image.
But a dataframe isn't enough - let's bring this to life with visuals.
df.plot.bar(x='hex',
y='count',
rot=0,
color=df['hex'].tolist(),
xlabel='Colors',
ylabel='Color Frequency',
title='Most Common Colors in Image',
figsize=(16, 8))
<AxesSubplot:title={'center':'Most Common Colors in Image'}, xlabel='Colors', ylabel='Color Frequency'>
X = np.hsplit(ar, np.array([1, 2]))[0].flatten().tolist()
Y = np.hsplit(ar, np.array([1, 2]))[1].flatten().tolist()
Z = np.hsplit(ar, np.array([1, 2]))[2].flatten().tolist()
fig = plt.figure(figsize=(16, 16))
ax = fig.add_subplot(111, projection = '3d')
ax.scatter(X, Y, Z, s=0.1, c=ar / 255.0)
plt.show()
Voila!¶
We've now taught a machine to perceive color like we do.
We've taught it to look past the noise and see the patterns within the image, and report those colors to us in an intuitive and logical way.
But how do we scale this? And how do we utilize this data to perform a new type of analysis?¶
We have art preserved that is centuries old.
We know who painted it, where it was created, and when.
Do patterns of dominant and secondary colors diffuse through time and space?
Perhaps, much like a scientific discovery, the use of colors in art arises in a certain place, and spreads its influence to neighboring areas.
And if this diffusion exists as a pattern, then it could be assessed analytically.
This is the question that our group – Tylor Mondloch, Nigel Joseph, and Stephen Zhu – will address.
We’ll begin by extracting images and their associated metadata from the following institutions:¶
Budget Collector Inc, Art Institute of Chicago, Museum of Modern Art (MoMA), and Rijksmuseum.
Artwork with defined place and time of creation will be preferentially selected.
We'll then run each of these images through the color clustering algorithm.
Thus, we'll create a dataset containing the most dominant colors for each painting, along with their associated metadata.
Once this data is assembled...¶
The door is open for both time series and spatial regression.
Does the prevalence of certain colors nearby and recently predict their use here and now?
If a color scheme is popular for long enough, does it reach the end of its shelf life and become obsolete?
Which color patterns succeed, and which ones fail? Furthermore, what do successful color patterns have in common?
These are the sort of questions...¶
that we aim to answer, and it all starts with recognizing how humans perceive color, and how we can train a machine to emulate that perception.
Meet Team: Analytics Avengers
Mr. Mondloch is a data scientist, working with a Big 4 Consulting Firm. His day-to-day includes building statistical models for cybersecurity contracts. He was born in South Dakota and now resides in Billings, Montana.
Stephen is a data scientist working with a hydro electricity company. In his spare time, he loves rock climbing and modeling the financial market. He was born in Hangzhou, China, and now resides in Vancouver, Canada.
Nigel is an analytics manager at a multinational pharmaceutical company, currently providing business insights and forecasting expertise for newly launched products. In his spare time, Nigel likes to play board games and golf (poorly).
Download the App
Inspired Me
“The app inspired an artistic part of me that’s always been there but hasn’t been tapped into for quite some time.”
Challenges Snobbish Ideals
“This innovative use of technology challenges the snobbish idea that only the rich can afford great art by helping anyone learn how to confidently and affordably bring the power of beauty into their lives.”
Discover Yourself
“It’s fun. It helps you discover something about yourself and gives you an idea of what speaks to you.”
