About a week ago, I experienced the most incredible AI demo since my first encounter with ChatGPT. Trust me, this is no exaggeration.

This post isn’t just about a cool tool; it also touches upon a crucial, unresolved issue in the world of Generative AI: handling those massive, complex data sets that are key to unlocking deeper value in many business and research areas.

Enter NotebookLM

I recently had a strategy brainstorming session with Angus Grundy (Angus is developing highly effective strategic frameworks; reach out to him if you’re looking to strategise/plan or think things through in any business or personal context. Insights guaranteed.). We recorded our 90-minute chat using Zoom’s AI assistant. Later, Angus used that transcript with Google’s experimental tool, NotebookLM.

On some levels, NotebookLM is a game-changer. You can upload extensive data sets (up to 50 files of 500,000 words each. That’s nearly 50 War and Peaces!) and interact with a chatbot in elaborate ways. You can ask highly specific questions and generate timelines, study guides, FAQs… all cross-referenced to your sources. You can then create notes from the chat results or add your own, shift focus between sources, manipulate it all, and more. You can even create, at the push of a button, an “audio overview,” where two AIs create a podcast-like discussion about the content. It’s as strange as it is delightful. But is it actually useful?

Putting it to the test: WhatsApp edition

To truly evaluate this tool, I needed a complex, large data set that I knew intimately. Something nuanced but more idiosyncratic than classic novels (which I could get off the net), something with the kind of human detail and subtlety I look for when doing the work. Enter my WhatsApp chats, where some relationships have unfolded over years, in large parts via text (I’d like to say “millennials, innit”, but in this sense of using tech, I’m one myself).

I created two “notebooks.” One based on an 18-month chat history, another on over four years from another relationship (split into four files to fit the limit. No regrets. Almost.).

The results were revealing. There was a noticeable degree of drift, meaning the AI’s understanding wasn’t always perfect, often with factual mistakes and false conjectures. This was more pronounced with the larger data set. So, while NotebookLM is groundbreaking for handling long sources, the inherent weaknesses of the technology remain. It’s far from foolproof.

The uncanny podcast

The “podcast” feature reflects the above but is also downright uncanny. The AIs’ emotional nuances in discussing the chats were so real that had someone played it for me a few years ago, claiming hackers had made it after enjoying my chats as if they were a reality show, I’d have believed them. However, each generation of the podcast showed different “drifts.” In one, the AI focused on my friend as if she were the sole protagonist (of a dialogue, but I don’t blame it — she’s a brilliant writer!). It switched attributions; it changed someone’s age. In another instance, it suggested that I was “spiralling out about my receding hairline,” 😅 Dear reader, on my honour: I was so young, and it happened so fast that even back then, I didn’t.

The potential & the concerns

If you’re a student trying to fake-read a bunch of novels, you’d be, to use the academic term, “royally f — ed”.

Yes, NotebookLM could be very useful for analysing focus groups and other research transcripts, as long as the original moderator is there to catch any errors.

But I’m worried about people throwing massive data sets into LLMs and being amazed by the results, likely because they aren’t deeply familiar with the data. It’s tempting to overlook the failings because it’s so quick, easy and cheap (not for the environment, but that’s another story).

The tech utopians will say: “but see how quickly it’s evolving!”

As a dyed-in-the-wool technorealist, I’m not even sure perfection is possible. As I often say about generative AI — it’s not a bug, it’s a feature.

The podcast: try it yourself

But the podcast? You must try it for yourself. Even if you download a classic novel you’re familiar with from Project Gutenberg and hear what the AI “podcasters” think of it.

But here’s something a bit different: I fed a notebook with my CV, Biog, client quotes, and a semi-random collection of my writing. This resulted in a “podcast” “overview” of much of “my thinking” through the years. And all those air quotes are because they sound 100% real and almost comically flattering. But listen closely, and it’s not a discussion of my work, but a discussion of a fruit salad of my work. A never-recorded recording of two enthusiastic people who don’t exist, having a discussion that never happened.

But hey, at least apparently, I’m “not afraid to ask the tough questions.”

Strange times, my friends, strange times.