How ChatGPT's Reasoning Model Helped Me Draft a Fantasy Hockey Team

September 26, 2024
|
By:
Matt Collette

AI Generated Summary:

  • Testing o1 in a Fantasy Hockey Draft: How good is OpenAi's new reasoning model and can it help marketing and communications teams? We put it to the test during a fantasy hockey draft, using detailed data on player stats, team dynamics, and league rules.
  • Multiple-step, Complex Reasoning on Display: o1 impressed by developing a tailored draft strategy based on multiple variables, offering strategic recommendations similar to how marketers can use AI to analyze customer segments, competitor behavior, or campaign performance.
  • Comparison to o1-preview, O1-mini and GPT-4o: What model is best for what? o1-mini excelled in real-time decisions, while o1-preview was stronger for strategic planning and deeper analysis. Its larger context window (128k tokens vs. GPT-4o’s 32k tokens) allowed it to handle more data and make more informed decisions.
  • Marketing and Communications Applications: What can we learn from a fantasy hockey draft? A lot as it turns out!

5 Things I Learned about the New Model’s Application to Marketing and Communications

Last week, I wrote about OpenAI’s shiny new reasoning model, o1, and hinted at how it could shake up how marketing and communications professionals use Generative AI. I was all set to dive into serious stuff to test the new model out —media reports, sales data, ad campaigns…..the usual—but then I thought, “Why not go for something that could make a real difference in the world?”.  Something with real stakes on the line. So, I put o1-Preview to the ultimate challenge: helping me draft my fantasy hockey team.

If o1 could help me crush a draft in a league of die-hard hockey fanatics, imagine what it could do for marketing strategy! Here’s how it went down and why I think it matters for marketers and communicators like you.


RECAP: WHAT IS O1?

o1-preview (because it is in preview mode) is OpenAI’s latest reasoning model, designed to go beyond the capabilities of GPT-4. While GPT-4 is brilliant at generating content and answering questions, o1 excels at solving complex, multi-step problems—where logic, decision-making, and strategy are crucial. For marketing and communications, this isn’t just about automating content creation but using Generative AI to shape strategies and help tackle higher-level decision-making.


My Fantasy Hockey League: The Basics

To provide some context, I play in a keeper fantasy hockey league. Here’s how it works:

  • Keeper Format: Each team keeps a core group of six players from year to year, making every draft strategic as teams fill the remaining roster spots (6 keepers + 9 drafted players for a total of 15 players per team).

  • Team Composition: Since each team has a different mix of keepers—some might retain more defensemen or goalies, while others favor forwards—the draft is less about picking the best available player overall and more about selecting the best player to complement your keepers.
  • Balancing Stat Categories: Players accumulate stats in categories like goals, assists, and penalty minutes. Teams earn points based on how they rank in each category. For example:
    • The team with the most penalty minutes gets the highest points in that category.
    • The team with the fewest penalty minutes gets the lowest points.

The objective is to lead in as many stats categories as you can and ultimately, become league champion.

Why This Was a Good Test of o1's Capabilities

I chose o1 for my fantasy hockey draft because it’s not just about making simple, one-off decisions—it’s about navigating a multi-step, strategic process with multiple variables at play. A successful draft strategy requires analyzing a complex set of factors, and I wanted to see if o1 could handle that level of complexity.  

In this case, o1 had to:

  • Fill positional gaps: Whether I needed more defensemen, goalies, or forwards, o1 would need to factor in positional needs to ensure a balanced lineup.
  • Prioritize key categories: o1 would need to analyze my team’s strengths and weaknesses (hits, penalty minutes) and recommend players that could help improve those categories.
  • Anticipate competitors’ picks: With each team making selections based on their needs, o1 would need to predict what others would do and adjust the strategy accordingly.
  • Adapt to changes: During the draft, o1 would need to adjust the strategy and pick recommendations when players I wanted were taken by others, dynamically updating its recommendations in real-time.

Much like building a marketing strategy and executing a campaign, o1 would need to analyze layers of data and predict competitor behavior. If o1 could handle this level of complexity in real-time, I knew it had potential for more advanced use cases.

Plus, I came in third to last in my pool last year, so it's safe to say my drafting skills have room for improvement!!

The Setup: Feeding the Beast with Data

To prepare for the draft, I wanted to maximize o1's understanding of player performance. Within ChatGPT, I gave the o1 model an overwhelming amount of data:

  • 2022-2023 player stats for 250 players
  • 2023-2024 player projections for 250 players
  • Standard deviation data and team performance metrics for 250 players
  • Average draft rankings for the platform we use for 300 players
  • 2024-2025 projected stats for 300 players

I also shared information on my league’s setup, how points are awarded, the keepers of each of the other teams in the fantasy pool, and the unique roster positions each team had to fill.

Sample Standard Deviation Data for 2023-2024


The Strategy: o1 in Action

With all the data loaded, o1 went to work — and it didn’t disappoint! It quickly broke down my team's needs, identifying gaps in areas like right wing, defense, and penalty minutes, and prioritizing players to fill those weaknesses. It identified specific players I should pick for each round of the draft and alternates in case those players were selected by other teams.

ChatGPT o1-Preview Chain of Thought Details as it Breaks Down the Task and Prompt
ChatGPT o1-Preview Proposed Draft Strategy
The Results – How Did I Do?

This tailored strategy gave me a clear edge throughout the draft. Here’s how it played out:

  • Prioritizing Scarce Positions:  o1 emphasized the need to focus on right wingers and defense early due to my team’s gaps. Following its advice, I drafted Timo Meier (NJD - LW/RW) first, adding versatility and strength in goals, shots, and hits. Then, I secured Moritz Seider (DET - D), a defenseman who contributes in multiple categories, boosting my offense.
  • Balancing Stat Categories: o1 ensured I didn’t overinvest in any one stat category, guiding me to make well-rounded picks. It advised me to draft Alex DeBrincat (DET - LW/RW) to add versatility on the right wing. For defense, o1 recommended Shea Theodore (VGK - D), a player known for his strong assist potential and contributions in power-play points. Later in the draft, I added Cole Caufield (MTL - RW), a young winger with high upside in goals and power-play points.
  • Targeting High-Upside Players: Late in the draft, o1 recommended high-upside players like Jordan Kyrou (STL - C/RW), a multi-category contributor, and Pyotr Kochetkov (CAR - G), a young goalie with breakout potential.

In comparison to past drafts, o1’s strategic insights gave me a clearer path to building a competitive and well-rounded team. I’m feeling much more confident about my lineup heading into the season, and I expect these moves to give me a strong chance at improving my performance this year.

ChatGPT o1-Preview Draft Recommendations Round by Round. I wasn't able to make all these picks, but stuck pretty close. As the draft evolved, ChatGPT o1-mini updated its recommendations based on the draft strategy.
What Did I Learn? The AI Draft Debrief

While it’s too early to say if o1's strategy will lead to a championship season (ask me in a few months), the process revealed five key insights:

  1. o1’s Reasoning Power Is No Jokeo1-preview impressed me with its ability to create a detailed draft strategy based on a myriad of stats. It didn’t just spit out generic player rankings—it analyzed my weaknesses (like PIMs and Hits) and recommended picks to address those gaps. This kind of high-level reasoning can be a game-changer for marketers too, especially when dealing with complex data sets like customer segments or multi-channel campaigns.

  2. Context Windows and Memory Matter – This experiment once again highlighted the importance of memory, context windows, and attention for Generative AI models.  

    With the large dataset I provided, o1 occasionally struggled to retain all the necessary details, which required me to periodically refresh or re-enter information. I would have been better off considering and managing what the model “knows” (data up until end of 2023) and what it doesn’t (new player data since its cutoff window) to maximize data inputs and management.

    o1-preview offers a significant advantage with its larger context window compared to GPT-4o. It can handle 128k tokens (100,000 words), while GPT-4o processes up to 32k tokens (26,000 words). As you use o1 and GPT-4o, the difference in these context windows becomes obvious, allowing for more refined outputs based on your prompt as your chat with the model evolves (e.g. ChatGPT forgets less things you’ve told it with o1 versus 4o).  

    When it comes to Generative AI, not only do we need to see improvements in the models, but we need more compute power to handle more memory, larger context windows, attention in addition to inference.  The larger the capacity for these elements, the greater the improvements we will see in productivity.  

  3. Real-Time Decision Making Potential o1 wasn’t just a pre-draft tool; it helped me make picks as the draft unfolded. Its ability to analyze the real-time flow of the draft and keep me on strategy was invaluable—just as real-time data analysis is in marketing, whether you’re tweaking an ad campaign based on live engagement data or adjusting a social strategy mid-launch.

  4. o1 vs. o1 Mini: Both Have Their Place - I alternated between o1-preview and o1-mini during the draft to manage prompt usage and maximize efficiency.  

    o1-preview excelled with the overall draft strategy — its deep reasoning abilities were ideal for this more complex, multi-step challenge where deeper chain of thought reasoning was a plus.  

    o1-mini was great during the draft for real-time analysis and pick recommendations. o1-mini proved faster and more efficient in the moment when I was on the clock, generating quick insights based on the draft strategy.

    For marketing and communications professionals, this experience highlights an essential takeaway when using these reasoning models – the importance of balancing the complexity of the tas with the speed of response and cost of using the different models.

  5. Overkill for Simple Tasks - After the draft, I asked o1 to help me summarize the experience for this article.  It became clear that o1’s advanced reasoning was overkill for low-cognition tasks like basic content creation. While it excelled in complex problem-solving during the draft, its "chain-of-thought" reasoning made the output unnecessarily detailed, with a lot of over-explaining and rationalization —almost like it was overthinking my request.  Stick to the base models for such things.
Post-Draft: Trade Recommendations and Future Strategy

While the draft went really well, I was not able to address all the deficiencies on my team. I may have to do so via a trade with another team in the pool. o1 helped here as well, assessing potential trades and trade partners.

In the context of marketing or comms, Generative AI can assess gaps in a campaign or audience engagement (like gaps in my fantasy roster) and recommend strategies (similar to suggesting trades) to reallocate resources—whether that’s changing the messaging, focusing on a different demographic, or adjusting ad spend to improve overall performance.

ChatGPT o1-Preview Suggested Trade Partners and Players
Marketing & Communications Tie-In: What Can o1 Teach Us?

So, what does my fantasy hockey draft have to do with marketing and communications? A lot, as it turns out:

  • Strategic Insights – Just as o1 identified gaps in my roster and recommended targeted picks to fill them, Generative AI in marketing can do the same by identifying audience pain points or market opportunities. Imagine using o1 to analyze customer behavior, competitor activity, or even regional trends.
  • Real-Time Adaptation: One of the most impressive features of o1 was its ability to adapt in real-time. As players I wanted were drafted by competitors, o1 recalibrated and provided alternative picks on the fly. Similarly, in communications, Generative AI-powered real-time feedback can be invaluable when a campaign needs quick adjustments. Whether it’s pivoting key messaging due to public response and media traction, or a social media post based on live data, more sophisticated models can help you react faster.
  • Contextual Awareness and Personalization: o1 wasn’t just looking at isolated stats; it analyzed my entire team’s makeup and needs within the context of the league. In marketing, Generative AI can take the same approach by examining individual customer journeys, segment-specific engagement patterns, or even the larger competitive landscape.
  • Efficiency vs. Complexity: One of my key takeaways from using o1 was learning when to apply the right level of Generative AI power. o1 was fantastic for strategic planning, while o1-Mini was better for quick, real-time recommendations. The same applies to marketing: not every task requires deep cognitive AI. For simpler decisions like A/B testing or routine copy generation, leaner models might be faster and more cost-effective, whereas complex campaign strategy or customer segmentation may benefit from full-powered AI models.

In short, just as o1 guided me through a successful draft, Generative AI can offer marketers and communicators the same level of detailed analysis, personalized strategy, and real-time adjustment capabilities.

Final Thoughts

At Sequencr, we talk a lot about how Generative AI is reshaping marketing and communications, enabling faster, smarter decisions. Tools like o1 bring this concept to life by turning complex data into actionable insights. While it may be too early to tell if o1 will win me a fantasy championship, its performance in my draft shows how AI can help marketers make smarter, more strategic decisions.