lexffe.xyzmy little tech corner of the internetZola2026-04-01T00:00:00+00:00https://lexffe.xyz/atom.xmlGoing through my Spotify history2026-04-01T00:00:00+00:002026-04-01T00:00:00+00:00
Unknown
https://lexffe.xyz/tech-blog/spotify-history/<h1 id="premise">Premise</h1>
<p>I am a giant music nerd and recently I am going through a reminiscing nostalgic trip. I want to know what I used to listen. I found out that Spotify allows you to request a copy of your extended streaming history.</p>
<p>(It is impossible to know my listening history before Spotify. 8 year old me relied on family for music and 12 year old me used to play music on YouTube and Google Music (RIP). I requested a copy of my YouTube data using takeout but none of them are useful (they only preserve 1 year of history it seems.))</p>
<h1 id="the-data">The data</h1>
<p>The data came in a zip file. Inside is a README PDF that contains descriptions of the columns.
The actual dataset spans through multiple JSON files, each around 12.8MB big.</p>
<p>An example of a data point may look like this:</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#C6D0F5, #CDD6F4); background-color: light-dark(#303446, #1E1E2E);"><code data-lang="javascript"><span class="giallo-l"><span>[</span></span>
<span class="giallo-l"><span style="color: light-dark(#949CBB, #9399B2);"> {</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">ts</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">2020-04-11T00:27:53Z</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">platform</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">iOS 13.4 (iPhone11,8)</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">ms_played</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#EF9F76, #FAB387);"> 174674</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">conn_country</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">SG</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">ip_addr</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">203.0.113.x</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">master_metadata_track_name</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">Jack To The Future (feat. Russoul)</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">master_metadata_album_artist_name</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">Alex Metric</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">master_metadata_album_album_name</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">Jack To The Future (feat. Russoul)</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">spotify_track_uri</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">spotify:track:74cksX4vgcm0cHKaV7XE2L</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">episode_name</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">episode_show_name</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">spotify_episode_uri</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">audiobook_title</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">audiobook_uri</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">audiobook_chapter_uri</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">audiobook_chapter_title</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">reason_start</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">trackdone</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">reason_end</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">trackdone</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">shuffle</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#EF9F76, #FAB387);"> false</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">skipped</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#EF9F76, #FAB387);"> false</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">offline</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#EF9F76, #FAB387);"> false</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">offline_timestamp</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> null</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#A6D189, #A6E3A1);"> "</span><span style="color: light-dark(#A6D189, #A6E3A1);">incognito_mode</span><span style="color: light-dark(#A6D189, #A6E3A1);">"</span><span style="color: light-dark(#81C8BE, #94E2D5);">:</span><span style="color: light-dark(#EF9F76, #FAB387);"> false</span></span>
<span class="giallo-l"><span style="color: light-dark(#949CBB, #9399B2);"> }</span><span style="color: light-dark(#949CBB, #9399B2);">,</span></span>
<span class="giallo-l"><span style="color: light-dark(#949CBB, #9399B2);font-style: italic;"> //</span><span style="color: light-dark(#949CBB, #9399B2);font-style: italic;"> ...</span></span>
<span class="giallo-l"><span>]</span></span></code></pre>
<p>A few interesting things:</p>
<ul>
<li>The IP address can be IPv4 and IPv6. A potential normalisation might be using IPv6's IPv4-Mapped address prefix like <code>::ffff:1.2.3.4</code>. The database may already have types to deal with IP addresses though.</li>
<li>The IP address may get anonymised at the Spotify side. The last 8-bit of the subnet is replaced with a <code>x</code>.
<ul>
<li>You can still tell which ISP you were on by using <code>bgp.he.net</code>.</li>
</ul>
</li>
<li>Spotify is being smart and gave us the album artist's name. Makes sense because there could be a featuring artist on the track, or it could be a compilation album.</li>
</ul>
<h1 id="quack">"Quack"</h1>
<p>Even though it is being given to me in JSON, this looks like tabular data to me. The only reasonable move
is to put it into a database and query it instead of using grep.</p>
<p>For the data size (around 640k rows), I could have also just used SQLite with a CLI but it really isn't suitable for analytic workloads, especially if I am enriching the data with more metadata from Spotify later, the joins alone in SQLite would slow down the queries a lot.</p>
<p>At $work we use ClickHouse for large scale data analysis. I am very much used to ClickHouse SQL syntax.
But if this is a personal project, why not use this opportunity to learn a new technology?</p>
<p>I turned to DuckDB as I want a database that's as easy to use as SQLite but can handle the data I gave it.</p>
<p>I ingested the data into DuckDB with a tiny bit of normalisation around the IP addresses and I skipped all the
audiobook stuff because I never used it, as well as the podcast stuff.</p>
<h2 id="file-size">File size</h2>
<table><thead><tr><th>Format</th><th>Size (MB)</th></tr></thead><tbody>
<tr><td>JSON</td><td>513</td></tr>
<tr><td>DuckDB</td><td>38</td></tr>
<tr><td>Arrow</td><td>53.2</td></tr>
<tr><td>Parquet</td><td>34.1</td></tr>
</tbody></table>
<p>Even with compression, my dataset still weighs 34.1MB. I did some optimisation such as decoding the spotify track URI's base62 ID component into blobs and store the blobs, and normalisations like storing tracks in a different table and create a view to join the history table with the track table back together. This took the file size down to 21.8MB. I guess I can further normalise by having the user agent in a different table / enum, storing <code>reason_start</code> and <code>reason_end</code> as enums, but that's an exercise for later.</p>
<h1 id="excerpt-on-what-i-found">Excerpt on what I found</h1>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#C6D0F5, #CDD6F4); background-color: light-dark(#303446, #1E1E2E);"><code data-lang="sql"><span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">select</span><span> </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);"> distinct</span><span> artist_name,</span></span>
<span class="giallo-l"><span style="color: light-dark(#8CAAEE, #89B4FA);font-style: italic;"> count</span><span>(</span><span>) </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">as</span><span> count </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">FROM</span><span> spotify_history_view </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">WHERE</span><span> ts </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">between</span><span style="color: light-dark(#A6D189, #A6E3A1);"> '</span><span style="color: light-dark(#A6D189, #A6E3A1);">2020-01-01</span><span style="color: light-dark(#A6D189, #A6E3A1);">'</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> and</span><span style="color: light-dark(#A6D189, #A6E3A1);"> '</span><span style="color: light-dark(#A6D189, #A6E3A1);">2026-04-01</span><span style="color: light-dark(#A6D189, #A6E3A1);">'</span><span> </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">GROUP BY</span><span> artist_name </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">ORDER BY</span><span> count </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">desc</span><span> </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">limit</span><span style="color: light-dark(#EF9F76, #FAB387);"> 10</span><span>;</span></span></code></pre>
<p>My top 10 artists between 2020 and now are</p>
<ul>
<li>deadmau5,</li>
<li>Flume,</li>
<li>Skrillex,</li>
<li>Bring Me The Horizon,</li>
<li>Lana Del Rey,</li>
<li>London Grammar,</li>
<li>Kendrick Lamar,</li>
<li>Radiohead,</li>
<li>ODESZA, and</li>
<li>The Prodigy.</li>
</ul>
<p>Not surprising. But when did I actually start listening to them?</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#C6D0F5, #CDD6F4); background-color: light-dark(#303446, #1E1E2E);"><code data-lang="sql"><span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">with</span><span> top_artists </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">AS</span><span> (</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">select</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);"> distinct</span><span> artist_name,</span></span>
<span class="giallo-l"><span style="color: light-dark(#8CAAEE, #89B4FA);font-style: italic;"> count</span><span>(</span><span>) </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">as</span><span> count,</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">FROM</span><span> spotify_history_view</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">WHERE</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> timestamp</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> between</span><span style="color: light-dark(#A6D189, #A6E3A1);"> '</span><span style="color: light-dark(#A6D189, #A6E3A1);">2020-01-01</span><span style="color: light-dark(#A6D189, #A6E3A1);">'</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> and</span><span style="color: light-dark(#A6D189, #A6E3A1);"> '</span><span style="color: light-dark(#A6D189, #A6E3A1);">2026-04-01</span><span style="color: light-dark(#A6D189, #A6E3A1);">'</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">GROUP BY</span><span> artist_name</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">ORDER BY</span><span> count </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">desc</span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">limit</span><span style="color: light-dark(#EF9F76, #FAB387);"> 10</span></span>
<span class="giallo-l"><span>) </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">SELECT</span><span> </span></span>
<span class="giallo-l"><span>artist_name, </span><span style="color: light-dark(#8CAAEE, #89B4FA);font-style: italic;">min</span><span>(</span><span style="color: light-dark(#CA9EE6, #CBA6F7);">timestamp</span><span>) </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">from</span><span> spotify_history_view </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">JOIN</span><span> top_artists </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">using</span><span> (artist_name) </span></span>
<span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">group by</span><span> artist_name;</span></span></code></pre><table><thead><tr><th>artist_name</th><th>min("timestamp")</th></tr></thead><tbody>
<tr><td>Lana Del Rey</td><td>2014-03-17 06:44:11</td></tr>
<tr><td>The Prodigy</td><td>2014-12-20 14:24:23</td></tr>
<tr><td>Bring Me The Horizon</td><td>2015-02-14 14:45:11</td></tr>
<tr><td>London Grammar</td><td>2014-03-17 20:19:13</td></tr>
<tr><td>deadmau5</td><td>2014-04-14 21:42:46</td></tr>
<tr><td>Flume</td><td>2015-01-11 02:35:49</td></tr>
<tr><td>ODESZA</td><td>2014-12-30 01:52:43</td></tr>
<tr><td>Radiohead</td><td>2014-05-13 21:03:06</td></tr>
<tr><td>Kendrick Lamar</td><td>2015-01-17 23:39:01</td></tr>
<tr><td>Skrillex</td><td>2014-04-12 12:04:00</td></tr>
</tbody></table>
<p>Exciting. I have been listening to these artists for more than 10 years.</p>
<p>How about top albums? This is harder to answer with this dataset because there is no play count for each album so we can only estimate based on play counts for each tracks in each album. You can skew this data if you just play one song multiple times. (There might be a way but the SQL is too annoying to conjure.)</p>
<pre class="giallo" style="color-scheme: light dark; color: light-dark(#C6D0F5, #CDD6F4); background-color: light-dark(#303446, #1E1E2E);"><code data-lang="sql"><span class="giallo-l"><span style="color: light-dark(#CA9EE6, #CBA6F7);">select</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> date</span><span>(</span><span style="color: light-dark(#8CAAEE, #89B4FA);font-style: italic;">min</span><span>(</span><span style="color: light-dark(#CA9EE6, #CBA6F7);">timestamp</span><span>)) </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">as</span><span> ts_date, </span><span style="color: light-dark(#8CAAEE, #89B4FA);font-style: italic;">count</span><span>(</span><span>) </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">as</span><span> album_count, album_name, artist_name </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">from</span><span> spotify_history_view </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">group by</span><span> artist_name, album_name </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">order by</span><span> album_count </span><span style="color: light-dark(#CA9EE6, #CBA6F7);">desc</span><span style="color: light-dark(#CA9EE6, #CBA6F7);"> limit</span><span style="color: light-dark(#EF9F76, #FAB387);"> 20</span><span>;</span></span></code></pre><table><thead><tr><th>ts_date</th><th style="text-align: right">album_count</th><th>album_name</th><th>artist_name</th></tr></thead><tbody>
<tr><td>2019-03-21</td><td style="text-align: right">1225</td><td>Hi This Is Flume</td><td>Flume</td></tr>
<tr><td>2016-12-02</td><td style="text-align: right">1167</td><td>W:/2016ALBUM/</td><td>deadmau5</td></tr>
<tr><td>2017-06-04</td><td style="text-align: right">1098</td><td>hopeless fountain kingdom</td><td>Halsey</td></tr>
<tr><td>2014-03-17</td><td style="text-align: right">885</td><td>If You Wait</td><td>London Grammar</td></tr>
<tr><td>2015-09-13</td><td style="text-align: right">885</td><td>Oh Wonder</td><td>Oh Wonder</td></tr>
<tr><td>2014-05-01</td><td style="text-align: right">794</td><td>No Mythologies to Follow (Deluxe)</td><td>MØ</td></tr>
<tr><td>2015-08-03</td><td style="text-align: right">790</td><td>Down to Earth</td><td>Flight Facilities</td></tr>
<tr><td>2015-09-21</td><td style="text-align: right">787</td><td>BADLANDS</td><td>Halsey</td></tr>
<tr><td>2016-05-19</td><td style="text-align: right">772</td><td>The Life Of Pablo</td><td>Kanye West</td></tr>
<tr><td>2017-06-16</td><td style="text-align: right">771</td><td>Melodrama</td><td>Lorde</td></tr>
<tr><td>2015-01-20</td><td style="text-align: right">757</td><td>My Beautiful Dark Twisted Fantasy</td><td>Kanye West</td></tr>
<tr><td>2015-01-30</td><td style="text-align: right">741</td><td>while(1<2)</td><td>deadmau5</td></tr>
<tr><td>2014-03-08</td><td style="text-align: right">731</td><td>True</td><td>Avicii</td></tr>
<tr><td>2018-07-13</td><td style="text-align: right">710</td><td>where's the drop?</td><td>deadmau5</td></tr>
<tr><td>2014-03-17</td><td style="text-align: right">699</td><td>Settle</td><td>Disclosure</td></tr>
<tr><td>2014-03-31</td><td style="text-align: right">686</td><td>Days Are Gone</td><td>HAIM</td></tr>
<tr><td>2019-03-29</td><td style="text-align: right">679</td><td>WHEN WE ALL FALL ASLEEP, WHERE DO WE GO?</td><td>Billie Eilish</td></tr>
<tr><td>2017-06-07</td><td style="text-align: right">677</td><td>I See You</td><td>The xx</td></tr>
<tr><td>2014-04-15</td><td style="text-align: right">670</td><td>> Album Title Goes Here <</td><td>deadmau5</td></tr>
<tr><td>2016-09-16</td><td style="text-align: right">661</td><td>New Age | Dark Age (Deluxe Version)</td><td>Karma Fields</td></tr>
</tbody></table>
<h1 id="after-words">After words</h1>
<p>This dataset is basically Spotify Wrapped but better.</p>
<p>Spotify recently greatly reduced the amount of data they serve over the Web API.
They no longer serve metadata such as genres, BPM, etc.
My original plan was to write a function (either ClickHouse UDF or some JS) to fetch these metadata on-demand but that's no longer viable.</p>
<p>I might jot down the process of getting more metadata out of Spotify, MusicBrainz and Last.fm
in a future post. It is still in progress and involves setting up a MusicBrainz database locally, attempting to pglite and a bit of LLM wrangling. I also want to look into fuzzy text search.</p>
<p>I also have friends asking me how to make this. I might make a little client-side web app for people
to play with. DuckDB has a WASM version so this can all be done without a server.</p>
<p>In the mean time, I'd like to leave a toy app that generates wallpapers from album arts <a rel="external" href="https://album-art-wallpaper-gen.vercel.app/">here</a>. (<a rel="external" href="https://github.com/lexffe/album-art-wallpaper-gen">source</a>)</p>
<h2 id="n-b-i-want-to-share-this-but-nevermind">N.B. I want to share this but nevermind</h2>
<p>I had a toy site ready to go, that lets people go and query an anonymised version of the data, but
even then LLMs reminded me that AI giants and scrapers could just download the parquet file and
use it for commercial purposes. :/</p>
Curriculum Vitae2026-03-23T00:00:00+00:002026-03-23T00:00:00+00:00
Unknown
https://lexffe.xyz/cv/Tech I appreciate from the software engineering world for the past n years.2026-03-23T00:00:00+00:002026-03-23T00:00:00+00:00
Unknown
https://lexffe.xyz/tech-blog/things-i-appreciate/<p>Over time I see more and more negativity in the software engineering and computer science community. I myself become opinionated and polarised as well. But I realised that this is not good for anyone, especially in the mental health department.</p>
<p>For this post, I want to list out things that bring me joy when using it. It is not just things that I have used before but things I understand and see the potential.</p>
<h2 id="portable-simd-in-rust">Portable SIMD in Rust</h2>
<p>I think mechanical sympathy is very underrated. We spend so much time wielding code and sometimes we leave so much performance on the table. My ideal world is that the compiler is smart enough to do autovectorisation. Couple that with instruction level parallelism and out of order execution and your application would be flying. In general, if your code is laid out obvious enough, the compiler does tend to recognise patterns and autovectorise your code. But sometimes your code is just complicated enough, with enough for loops and if branches, that the compiler can’t figure out. Or sometimes you figured out how to calculate all the branches at once.</p>
<p>Recently at $work I have had the privilege and luxury to write high performance code. It is compute heavy and benefits heavily from vectorisation. Unfortunately I made the sin of optimising before measurement. The original code was more or less scalar but I did not check the emitted assembly nor did I do any profiling. So there was no way to tell if autovectorisation ever worked. That said, moving from scalar code to portable SIMD did yield a substantial boost. I did not use external libraries and used <code>std::simd</code>.</p>
<p>Things I appreciate:</p>
<ul>
<li>I have never really written SIMD code before so I pair cargo asm with Claude. This creates a nice feedback loop where you can guide optimisation based on the assembly.</li>
<li>The syntax and abstraction works. I look at <code>u32x4</code> and I understand what it means. I don’t need to remember mnemonics.</li>
<li>You can mix and match portable SIMD code with unsafe platform intrinsics. So if you see a codegen behaviour that you don’t think is efficient, then you can use code blocks guarded with platform detection and write with intrinsics.</li>
</ul>
<p>Things it could improve on:</p>
<ul>
<li>The codegen is generally really good, but kind of falls short on ARM targets. AVX codegen works better than NEON, specifically when it comes to swizzling. It is possible that it’s specific to my code. In the end, I prompted Claude to use the portable SIMD code to generate a NEON specific code block.</li>
<li>Portable SIMD is still in nightly...</li>
<li>If I build std during build-time as well (<code>-Zbuild-std</code>), it yields better assembly. I did not spend much time investigating this behaviour.</li>
</ul>
<p>(I could have picked C/C++ but I have not had good experience writing those before. Dealing with portable SIMD and/or intrinsics in C/C++ just isn’t on my wishlist. My wish is to write SIMD/vectorised code with Zig vectors in some other project.)</p>
<p>References:</p>
<ul>
<li><a rel="external" href="https://nnethercote.github.io/perf-book/introduction.html">https://nnethercote.github.io/perf-book/introduction.html</a></li>
<li><a rel="external" href="https://en.algorithmica.org">https://en.algorithmica.org</a></li>
</ul>
<h2 id="zig-cc">Zig CC</h2>
<p>Zig as a C compiler felt like the same GOARCH+GOOS magic I had. Combining that with cargo zigbuild and we are cross-compiling to every target. The only slight issue I have is that Zig target triples are not the same as the one in Rust.</p>
<h2 id="jujutsu">Jujutsu</h2>
<p>My workflow used to be:</p>
<ul>
<li>Checkout new branch</li>
<li>write code</li>
<li>Forgot to commit all the files so I commit all the files into a single commit</li>
<li>Painstakingly refactor the history so the commits make sense</li>
<li>Rebase to main</li>
<li>Colleague reviews my PR</li>
<li>Interactive rebase but pick that one commit that my colleague suggested changes and edits that commit</li>
<li>Stuck in rebase hell and painstakingly resolve all the conflicts</li>
<li>Force push</li>
</ul>
<p>Now, I am not saying that jujutsu doesn’t have the same steps, but my workflow has been way more pleasant recently. I started off by initialising jujutsu on the main codebase I work on, and I used VisualJJ to move files across changesets. I can put files into their changesets and reordering commits becomes trivial. I was not too happy that I have to open up VSCode just to do jj operations (because right now JJ has lackluster support in IntelliJ), so I moved over to jjui and jj cli. It took me a second to understand changesets (commits) and bookmarks (branches) but it feels so natural now.</p>
<p>I can also prompt Claude Code to create a changeset every time it does something, so I can see the changes individually.</p>
<h2 id="fast-tooling">Fast tooling</h2>
<p>People used to write tooling for scripted languages in that specific language. Gulp, babel, webpack were written in JS. Poetry, pipenv, conda were written in Python.<br />
We are definitely seeing a surge of people (re)writing tooling to improve developer experience. UV, TypeScript compiler Go rewrite, esbuild, ripgrep, oxc, are all good examples. Now, I have yet to use any of these tools except uv, but it is really nice to see that there are groups of people dedicating their time to make DX nicer for other devs. UV is now my preferred tool to work with Python.</p>
<h2 id="gpu-accelerated-ui">GPU accelerated UI</h2>
<p>I recently moved from VSCode to Zed and it feels really fast. I don’t have any scientific measurements that can back my claim but clicking around and typing feels responsive. It’s the power of Emacs/Neovim responsiveness but in a GUI. Ghostty boasts a GPU accelerated terminal as well, just like kitty, alacritty, wezterm, et cetera. I use ghostty mainly for the drop down terminal on macOS but the GPU acceleration is a nice bonus. (https://github.com/ghostty-org/ghostty#competitive-performance)</p>
<p>It makes so much sense. Why waste the CPU cycles on drawing glyphs in the terminal, when you could offload that to the GPU?</p>
<p>(Although, for heavy coding I am still using the JetBrains suite, which even though it uses way more resources, still manages to be responsive. I guess this is years of optimisation with Java/Swing?)</p>
<h2 id="webassembly">WebAssembly (?)</h2>
<p>My friend reminded me that we were talking about WebAssembly around 10 years ago (we were 17 at the time…). Unfortunately I do not have the chance to use WASM consciously in my day to day life but this might be the original Java vision. Instead of going and downloading a JRE / .NET runtime, you just have a browser, which most consumer electronic devices in the world have.</p>
<p>People are using WASM at the server-side as well! Again, I haven’t had the chance of using it but it seems like a good alternative to container based solutions. You can even run WASM/WASI on kubernetes.</p>
<h2 id="efi-and-efi-stub">EFI and EFI stub</h2>
<p>I had bad memories when I was messing around dual booting Windows and Linux when I was a kid, messing with the bootloader and disk sectors so BIOS could find the bootloader. Grub rescue is really not something I know how to deal with.</p>
<p>Booting the Linux kernel directly from UEFI feels like magic. There’s no bootloader anymore and there’s one less thing to go through when booting a computer. (When Chrome OS first came out, I kept going back to this video about <a rel="external" href="https://www.youtube.com/watch?v=mTFfl7AjNfI">Chromium OS Fast Boot</a>.</p>
<p>Even though my home server uses Grub nowadays, it’s a nifty knowledge.</p>
<h2 id="kubernetes-and-infrastructure-as-code">Kubernetes (and Infrastructure as Code)</h2>
<p>This section might deserve its own post. I used to see k8s as this extra thing you need to run on top of IaaS but now I see the potential. I don’t even think that the control plane reconciliation with your desired state (GitOps or whatever) is the main feature. K8s provided a single API and abstraction for the big clouds. Every cloud has their idiom of doing things on their platform, but k8s is just this one standard and big clouds implement it. I can declare a deployment, a persistent volume and a load balancer and the big cloud translates it to their parlance. Pair that with terraform and you<br />
Of course when you go into minute details this argument tends to break down (IAM is cloud specific, PostgresOperator != managed DB, object storage, etc.), but this taught me that Kubernetes can be a powerful abstraction.</p>
<h2 id="sqlite">SQLite</h2>
<p>Nothing much to say other than it is my default flat-file database that just works.</p>