Core

Traditional software has bugs. LLM software has behaviors.

You don't test behaviors - you observe them, measure them, and guide them. This requires a fundamentally different approach.

Three Modes of Observation

Mode 1: Introspection The model evaluates itself during generation. Like a writer reviewing their draft before hitting send. Built-in confidence scoring, explanation generation, self-critique loops. This happens at inference time, costs tokens, but provides immediate guardrails.

Mode 2: Instrumentation Automated measurement of what actually happened. Response times, token usage, embedding similarities, retrieval success rates. Like application performance monitoring but for AI quality. Runs continuously, at scale.

Mode 3: Interpretation Humans make sense of what matters. Not everything can be measured - sometimes you need human judgment on tone, helpfulness, or edge cases. Expensive but irreplaceable for subjective quality and ground truth.

What This Looks Like in Practice

During Development:

• Start with exploration, not test cases - explore what your model can do

• Capture interesting behaviors as you discover them (the good, bad, and weird)

• Write expectations, not assertions: "should be empathetic" not "must contain 'sorry'"

• Build your eval suite backwards - get it working first, then define what "good" means based on real outputs

During Deployment:

• Shadow scoring on real traffic before switching

• Gradual rollout based on confidence thresholds

• Automatic fallback to previous version if scores drop

• No binary deploy/rollback - it's a confidence dial

During Operation:

• Stream of observations, not error logs

• Distribution of scores, not pass/fail counts

• Anomaly detection, not threshold alerts

• Behavioral drift tracking, not uptime monitoring

During Improvement:

• Human labels on strategic samples, not random QA

• Disagreement cases between evaluators get priority

• Corrections become test cases automatically

• Fine-tuning happens on production-validated examples

Embracing LLM Constraints

Embrace Uncertainty: Stop pretending LLMs are deterministic. Design for confidence intervals, not binary outcomes.

Compose, Don't Compile: Evaluators should be Lego blocks - mix and match for your use case. Need safety + factuality + tone? Stack them.

Optimize for Learning: Every evaluation should make the system smarter. If it doesn't feed back into improvement, why measure it?

Human Time is Sacred: Only escalate to humans when machines disagree or confidence is low. Make their input count by turning it into reusable signals.

(BTW, we are hiring!)

I hop you find this helpful:

Zero-shot Prompting

Zero-shot prompting allows you to leverage an LLM's capabilities by giving it task instructions without any examples or training data.

The power of zero-shot prompting lies in its simplicity—you communicate directly with the model using natural language instructions. For example, instead of showing multiple examples of sentiment analysis, you can simply ask:

Text: i'll bet the video game is a lot more fun than the film.
Sentiment:

Zero-shot prompting works because LLMs have been trained on vast amounts of data and can understand task contexts from clear instructions alone.

When to Use Zero-shot Prompting

You should consider zero-shot prompting when:

• You need quick results for general tasks

• The task is relatively straightforward and commonly understood

• You don't have example data readily available

• You want to test the model's base capabilities

• The task doesn't require complex reasoning or multi-step processes

Benefits and Limitations

Benefits:

• Simplicity in implementation

• No need to curate examples

• Faster execution without example processing

• Tests the model's true understanding of tasks

• Useful for rapid prototyping

Limitations:

• May be less accurate for complex or specialized tasks

• Performance can vary significantly between different models

• Less control over the exact format of outputs

• May struggle with nuanced or domain-specific tasks

For more complex tasks that require reasoning or specialized knowledge, you might need to consider other techniques like few-shot prompting or chain-of-thought prompting. However, zero-shot prompting serves as an excellent starting point for many applications and can often surprise you with its effectiveness.

To learn more about zero-shot prompting and its applications, you can explore the detailed research and documentation in Lil'Log's Prompt Engineering Guide or dive deeper into implementation strategies at promptingguide.ai.

Few-shot Prompting

Few-shot prompting involves providing the language model with examples of the desired task, helping it understand your intentions before handling new cases.

Implementation and Best Practices

When implementing few-shot prompting, you'll want to structure your prompt with clear example pairs. Here's a basic format:

Input: [First example input]
Output: [First example output]

Input: [Second example input]
Output: [Second example output]

Input: [Your actual input]
Output:

The effectiveness of few-shot prompting heavily depends on how you select and present your examples. Research has shown that choosing semantically similar examples to your target task can significantly improve performance. You can use embedding-based techniques like k-NN clustering to find relevant examples from your dataset.

Example Selection Strategy

When selecting examples, consider these key factors:

• Choose diverse examples that cover different aspects of the task

• Ensure examples are representative of the desired output format

• Keep the number of examples balanced with your token limit

• Maintain consistent formatting across all examples

Remember that the order of your examples matters. Studies have identified several biases that can affect model performance, including majority label bias (where the model favors more frequently seen labels) and recency bias (where it tends to favor the most recent examples).

Benefits and Trade-offs

Few-shot prompting offers several advantages over zero-shot approaches:

• Provides clearer context for complex tasks

• Helps establish consistent output formatting

• Reduces ambiguity in task interpretation

• Improves performance on specialized or domain-specific tasks

However, these benefits come with trade-offs. Each example consumes tokens from your context window, which can be particularly challenging when working with longer inputs or outputs. You'll need to balance the number of examples against your model's context length limits and cost considerations.

For optimal results, consider implementing dynamic example selection based on your input. Dynamic example selection, while more complex, can help maximize the relevance of your examples while minimizing token usage. Some researchers have even explored using Q-Learning and active learning techniques to optimize example selection for specific use cases.

Chain-of-Thought Prompting

Chain-of-Thought (CoT) prompting enhances the reasoning capabilities of large language models by encouraging them to break down complex problems into logical steps.

When to Use Chain-of-Thought

CoT prompting is particularly effective for tasks requiring multi-step reasoning, such as:

• Mathematical word problems

• Complex logical deductions

• Symbolic reasoning tasks

• Decision-making scenarios requiring multiple considerations

The technique works best with larger language models (>100B parameters), as they can better leverage the structured reasoning approach. For simpler tasks or smaller models, traditional prompting might be more appropriate.

Implementation and Examples

Here's how to implement CoT prompting effectively:

Q: John has 10 apples. He gives away 4 and then receives 5 more. How many apples does he have?

A: Let's solve this step by step:
1. John starts with 10 apples
2. He gives away 4, so 10 - 4 = 6 apples
3. He receives 5 more, so 6 + 5 = 11 apples
Therefore, John has 11 apples.

Q: [Your complex question]
A: Let's solve this step by step:
[Your reasoning steps]

Research from Wei et al. shows that formatting matters: using newline breaks between reasoning steps performs better than using periods or semicolons. Additionally, using "Question:" instead of "Q:" can improve performance.

Best Practices and Limitations

To maximize CoT effectiveness:

• Break down complex problems into clear, sequential steps

• Use explicit reasoning markers ("First," "Then," "Therefore")

• Include diverse examples in your prompts for better generalization

• Consider using self-consistency sampling to improve accuracy by generating multiple reasoning paths

However, be aware that CoT has limitations:

• Less effective with smaller models, which may produce incoherent reasoning

• Can sometimes generate plausible-sounding but incorrect reasoning chains

• May not improve performance on simple tasks that don't require multi-step thinking

For complex reasoning tasks, you can enhance CoT further by combining it with other techniques like self-consistency sampling, which generates multiple reasoning paths and selects the most consistent answer through majority voting.

Meta Prompting

Meta prompting involves guiding the LLM to generate and optimize prompts itself by providing higher-level instructions.

How Meta Prompting Works

When using meta prompting, you essentially create a prompt about prompts. For example:

Generate a prompt that would help extract key financial metrics from quarterly reports. The prompt should:

• Focus on specific numerical indicators

• Include validation steps

• Request structured output in JSON format

The LLM will then generate a task-specific prompt that meets these requirements, which you can use for your actual data processing.

Key Applications

Meta prompting is particularly valuable in scenarios where:

• You need to handle varying input formats or contexts

• The task requirements might change over time

• You want to automatically optimize prompts based on results

• You're building systems that need to adapt to different user needs

For instance, in a document analysis pipeline, you might use meta prompting to generate specialized extraction prompts based on the document type:

Input: Create a prompt for analyzing {document_type}
Context: The system needs to identify {key_elements}
Requirements:

* Maintain consistent output structure
* Include error handling
* Focus on {specific_metrics}

Dynamic Prompt Generation

One of the most powerful applications is dynamic prompt generation. Instead of using static prompts, you can create meta-level instructions that help the model adjust its approach based on the input:

Based on the user's technical expertise level ({expertise}), generate a prompt that will:

1. Explain {concept} at the appropriate depth
2. Use relevant examples for their field
3. Include follow-up questions tailored to their background

Benefits and Considerations

Meta prompting offers several advantages:

• Increased flexibility in handling diverse use cases

• Better adaptation to changing requirements

• More maintainable prompt management

• Improved scalability for complex applications

However, it's important to note that meta prompting adds a layer of complexity and may require more computational resources since you're essentially running two prompt cycles—one to generate the prompt and another to execute it.

When implementing meta prompting, focus on clear constraints and validation criteria to ensure the generated prompts align with your objectives and maintain consistent quality in the final outputs.

Self-Consistency

Self-consistency enhances the reliability of language models by generating multiple reasoning paths for the same problem and selecting the most consistent answer through majority voting.

When to Use Self-Consistency

You should consider implementing self-consistency when your tasks involve multi-step reasoning or when there are multiple valid approaches to reach a solution. According to research from Mercity AI, this technique has shown remarkable improvements across various benchmarks:

• 17.9% improvement on GSM8K (mathematical reasoning)

• 11.0% improvement on SVAMP (word problems)

• 12.2% improvement on AQuA (analytical reasoning)

The benefits become even more pronounced with larger language models, with improvements of up to 23% observed in models like LaMDA137B and GPT-3.

Implementation Approach

To implement self-consistency effectively:

1. Generate multiple solutions for the same problem using different reasoning paths.

2. Introduce randomness through various methods:

   • Altering the order of examples

   • Using model-generated rationales instead of human-written ones

   • Varying the complexity of reasoning chains

3. Aggregate the results through majority voting.

When working with training examples, you can follow the STaR (Self-Taught Reasoner) method as outlined by researchers like Lilian Weng:

• Generate reasoning chains and retain those leading to correct answers.

• Fine-tune the model with these generated rationales.

• Iterate until convergence.

Performance Considerations

Self-consistency is particularly effective because it's an unsupervised technique that requires no additional human annotation, training, or model fine-tuning. It remains robust across different sampling strategies and parameters, consistently enhancing performance.

For optimal results, consider implementing complexity-based consistency, where you explicitly prefer complex chains among all generations. This approach involves taking majority votes among only the top k complex chains, which has shown to be particularly effective in improving reasoning accuracy.

The technique becomes increasingly valuable as model size grows. Even for large models that already perform well, self-consistency consistently offers additional gains, with improvements of 12%-18% in accuracy on tasks like AQuA and GSM8K, even when using advanced models like PaLM-540B.

Generate Knowledge Prompting

Generate Knowledge Prompting is a powerful technique that helps LLMs perform better on tasks requiring deep contextual understanding by first generating relevant knowledge about a topic before attempting to answer questions or complete tasks.

How It Works

The technique follows a two-step process:

1. First, prompt the model to generate knowledge about the specific topic.

2. Then, use that generated knowledge to inform the final response.

Here's a basic template for knowledge generation:

Generate some knowledge about the input.

Examples:
Input: What type of water formation is formed by clouds?
Knowledge: Clouds are made of water vapor.

Input: {your_question}
Knowledge:

This initial knowledge generation step acts as an intermediate reasoning layer, allowing the model to explicitly state relevant facts and context before tackling the main task.

Implementation Steps

1. Knowledge Generation: Start by prompting the model to generate relevant knowledge about your topic. This creates an explicit knowledge base for the model to work with.

2. Knowledge Integration: Feed the generated knowledge back into the prompt along with your main task or question. This ensures the model has immediate access to relevant context.

3. Final Response: The model then uses both the generated knowledge and the original question to produce a more informed and accurate response.

Benefits and Applications

Generate Knowledge Prompting is particularly effective for:

• Complex questions requiring domain expertise

• Tasks involving temporal knowledge (historical or current events)

• Scenarios where implicit knowledge needs to be made explicit

• Cases where the model needs to demonstrate reasoning about specific facts

According to research by Liu et al., this technique has shown significant improvements in response quality even with just "internal retrieval"—generating knowledge without external sources. The method helps bridge the gap between the model's training data and the specific context needed for accurate responses.

This approach can be particularly powerful when combined with other techniques like Retrieval Augmented Generation (RAG) for external knowledge sources or Chain-of-Thought prompting for complex reasoning tasks. For instance, you can generate knowledge about multiple aspects of a problem, then use that knowledge to construct a step-by-step solution.

The technique is especially valuable in production environments where accuracy and reliability are crucial, as it provides an explicit trail of the knowledge being used to form responses, making it easier to verify and debug the model's reasoning process.

Prompt Chaining

Prompt chaining involves creating a sequence of prompts, where each prompt's output serves as input for the next, effectively breaking down complex tasks into manageable subtasks.

When to Use Prompt Chaining

You should consider implementing prompt chaining when your task:

• Requires multiple logical steps to complete

• Can be naturally broken down into smaller subtasks

• Needs intermediate validation or processing

• Would benefit from focused, specialized prompts rather than one large prompt

For example, instead of asking an LLM to analyze a long document and provide recommendations in a single prompt, you might chain prompts to first summarize the document, then identify key themes, and finally generate specific recommendations based on those themes.

Implementation Approach

To implement prompt chaining effectively:

1. Break down your complex task into discrete steps

2. Design specific prompts for each step

3. Create a pipeline where outputs flow as inputs

4. Include validation checks between steps

5. Handle errors and edge cases at each stage

The key is to make each prompt in the chain focused and specific, rather than trying to accomplish everything at once. This improves reliability and makes the system easier to debug and maintain.

Benefits of Prompt Chaining

Prompt chaining offers several advantages in production environments:

• Improved reliability: By breaking down complex tasks, each step becomes more manageable and reliable

• Better control: You can monitor and validate intermediate results

• Enhanced debugging: When issues occur, you can identify exactly which step in the chain failed

• Flexible architecture: Chains can be modified or extended without rebuilding the entire system

• Reusable components: Individual prompts in the chain can be reused across different workflows

Real-World Applications

In production systems, prompt chaining is particularly valuable for tasks like:

• Content generation: Breaking down the process into research, outlining, writing, and editing steps

• Data analysis: Sequencing data cleaning, analysis, and insight generation

• Customer service: Routing queries through understanding, context gathering, and response generation

• Document processing: Implementing staged approaches for extraction, analysis, and summarization

By implementing prompt chaining, you create more robust and maintainable LLM applications that can handle complex tasks with greater reliability and control. The key is to thoughtfully design your chains to match your specific use case requirements while maintaining clear boundaries between each step in the process.

Tree of Thoughts

Tree of Thoughts (ToT) enables language models to explore multiple potential solutions simultaneously through a branching tree structure, extending beyond the linear reasoning of Chain-of-Thought.

How ToT Works

ToT breaks down problems into coherent units of text called "thoughts" that serve as intermediate steps. At each step, the model generates multiple possible thoughts, creating branches in a tree-like structure. These branches are then explored using either breadth-first search (BFS) or depth-first search (DFS), with each state being evaluated through classifier prompts or majority voting.

The key innovation is that ToT allows language models to:

• Make deliberate decisions between multiple options

• Explore different reasoning paths simultaneously

• Self-evaluate choices at each step

• Look ahead or backtrack when needed

• Make globally optimal decisions

Performance Benefits

The effectiveness of ToT is particularly evident in complex problem-solving tasks. For instance, in the Game of 24, ToT achieved a 74% success rate when considering five possible solutions at each step (b=5), compared to just 7.3% for standard input-output approaches. In broader testing across 100 diverse tasks, ToT consistently outperformed traditional methods, achieving an average GPT-4 score of 7.56, compared to 6.19 for standard input-output and 6.93 for Chain-of-Thought approaches.

When to Use ToT

ToT is particularly effective for:

• Problems requiring non-trivial planning or search

• Tasks with multiple valid solution paths

• Scenarios where initial approaches might lead to dead ends

• Complex reasoning that benefits from exploring alternatives

• Creative tasks requiring evaluation of different possibilities

The technique shines in applications like creative writing, puzzle-solving, and complex mathematical problems where the ability to explore multiple pathways and backtrack when needed leads to more robust solutions.

For implementation, ToT can be integrated with various search strategies and evaluation methods. The breadth parameter (b) can be adjusted based on the complexity of the task—simpler problems might work well with b=1, while more complex ones benefit from higher values like b=5 to explore more possibilities simultaneously.

Retrieval Augmented Generation (RAG)

Retrieval Augmented Generation (RAG) enhances Large Language Models (LLMs) by combining document retrieval with answer generation, making it valuable when working with proprietary or dynamic data not part of the model's original training.

The process works in two distinct phases. First, the system retrieves relevant documents using dense embeddings—vector representations of both the query and potential source documents. Retrieval can be implemented using various database formats depending on your specific needs, including vector databases, summary indices, tree indices, or keyword table indices. The system identifies the most relevant documents by finding those whose vectors are closest to the query vector in terms of Euclidean distance.

In the second phase, the LLM generates a response by combining the user's query with the retrieved documents. This approach significantly improves the model's ability to provide accurate, factual responses while reducing hallucination, as it relies on retrieved facts rather than solely on its training data.

An advanced implementation of RAG is GraphRAG, developed by Microsoft Research, which extends the basic RAG architecture by incorporating knowledge graphs. This enhancement allows the system to connect disparate pieces of information, synthesize insights across multiple sources, understand summarized semantic concepts over large data collections, and combine access to both unstructured and structured data.

GraphRAG has shown particular effectiveness when working with complex datasets. For instance, it has been successfully applied to the Violent Incident Information from News Articles (VIINA) dataset, demonstrating significant improvements in generating comprehensive and diverse answers for complex analytical questions.

The key advantage of RAG is its ability to handle information that wasn't included in the model's initial training or fine-tuning phases. This makes it especially valuable for enterprise applications where you need to work with internal documentation, frequently updated information, proprietary data, and domain-specific knowledge.

When implementing RAG, you can leverage different types of storage systems based on your specific needs. The choice between vector databases, summary indices, or tree indices will depend on factors like your data structure, query patterns, and performance requirements.

Automatic Reasoning and Tool-use

Automatic reasoning and tool-use combines an LLM's reasoning capabilities with the ability to interact with external tools and APIs, enabling it to make decisions and take actions.

When to Use Automatic Reasoning and Tool-use

You should consider implementing automatic reasoning and tool-use when your application needs to:

• Solve complex problems that require multiple steps of logical reasoning

• Access external data or functionality not contained within the LLM's knowledge

• Generate action plans and execute them through external tools

• Automate workflows that combine decision-making with practical actions

Integration with External Tools

The implementation typically involves creating a framework where the LLM can:

1. Analyze the task and break it down into logical steps

2. Identify which tools or APIs are needed for each step

3. Generate the appropriate calls to these tools

4. Process the results and incorporate them into its reasoning chain

While specific implementations vary, the general pattern involves providing the LLM with:

• A description of available tools and their capabilities

• The format for tool invocation

• How to interpret and use tool responses

Applications and Benefits

This technique is particularly valuable in scenarios such as:

• Automated research workflows where the LLM needs to query databases or search engines

• Data analysis pipelines where the model must process information from multiple sources

• Task automation systems that need to interact with various APIs

• Decision support systems that combine reasoning with real-world data

The key advantage is the ability to extend the LLM's capabilities beyond its training data by connecting it to external tools and data sources. This creates more powerful and practical applications that can take real actions based on reasoned decisions.

Integration with Other Techniques

Automatic reasoning and tool-use often works in conjunction with other prompt engineering techniques. For example, it can be combined with ReAct (Reasoning and Acting) to create systems that can both reason about problems and take appropriate actions to solve them. This combination enhances the LLM's ability to handle complex tasks by breaking them down into manageable steps of reasoning and action.

The future development of this technique points toward more sophisticated frameworks that can automatically determine which tools to use and when, making AI systems more autonomous and capable of handling complex real-world tasks that require both reasoning and practical action.

Active-Prompt

Active-Prompt adapts and refines prompts in real-time based on the model's responses and performance, continuously optimizing output quality in production environments.

Dynamic Adaptation Mechanism

The core of Active-Prompt lies in its feedback loop system:

1. Initial prompt execution

2. Response evaluation

3. Prompt refinement based on performance metrics

4. Iterative improvement through continuous monitoring

This mechanism is particularly valuable when working with LLM applications that need to maintain high performance and reliability in production environments. The prompt adjustments can be based on various factors:

• Response quality metrics

• Context adherence

• Instruction following accuracy

• Error rates and completion success

• User interaction patterns

Implementation Approach

To implement Active-Prompt effectively, you'll need to:

1. Set up monitoring infrastructure to track prompt performance

2. Define clear evaluation metrics for response quality

3. Create adjustment rules for prompt modification

4. Implement feedback mechanisms for continuous improvement

Here's a conceptual example of how to structure an Active-Prompt system:

class ActivePromptSystem:
    def __init__(self, base_prompt, evaluation_metrics):
        self.current_prompt = base_prompt
        self.metrics = evaluation_metrics
        self.performance_history = []

    def evaluate_response(self, response):
        score = self.metrics.evaluate(response)
        self.performance_history.append(score)
        return score

    def adjust_prompt(self, score):
        if score < self.metrics.threshold:
            # Implement prompt refinement logic
            self.current_prompt = self.refine_prompt(score)
            
    def execute_with_monitoring(self, input_text):
        response = self.execute_prompt(self.current_prompt, input_text)
        score = self.evaluate_response(response)
        self.adjust_prompt(score)
        return response

Benefits for Production Systems

Active-Prompt offers several advantages for production LLM applications:

• Real-time quality maintenance through continuous monitoring

• Automatic adaptation to changing conditions or requirements

• Reduced need for manual prompt engineering interventions

• Improved reliability in production environments

• Better handling of edge cases through dynamic adjustments

The technique works particularly well when integrated with comprehensive monitoring systems that can track and analyze prompt performance metrics in real-time. This aligns with production requirements where maintaining consistent quality and reliability is crucial for mission-critical applications.

For optimal results, Active-Prompt should be combined with robust observability tools that can provide detailed insights into prompt performance and help identify areas needing adjustment. This enables both automated and manual refinements to the prompting strategy based on real-world usage patterns and performance data.

Directional Stimulus Prompting

Directional Stimulus Prompting (DSP) incorporates specific hints, cues, or keywords to guide the language model toward producing desired outputs in alignment with your goals.

How It Works

The core mechanism of DSP involves embedding strategic cues within your prompt that serve as "stimuli" for the model. These cues can take various forms:

• Keyword hints that should appear in the response

• Structural elements that guide the format

• Contextual signals that frame the desired perspective

• Specific terminology that should be incorporated

For example, instead of asking "Write about artificial intelligence," you might use DSP like this:

Write about artificial intelligence, incorporating these key aspects:
- Neural networks
- Machine learning algorithms
- Real-world applications
Focus on enterprise implementations and emphasize scalability.

The directional elements in this prompt ("enterprise implementations" and "scalability") help steer the model toward a specific type of response while the keyword hints ensure coverage of essential concepts.

When to Use DSP

DSP is particularly valuable when you need:

• Precise control over the model's output format

• Consistent inclusion of specific elements or terminology

• Alignment with domain-specific requirements

• Structured responses that follow particular patterns

• Outputs that maintain focus on certain aspects while avoiding others

This technique shines in professional contexts where output consistency and adherence to specific requirements are crucial. For instance, when generating technical documentation, product descriptions, or specialized reports where certain elements must be present in the final output.

Benefits and Considerations

The primary advantage of DSP is its ability to provide finer control over the model's outputs while maintaining natural language flow. Research indicates that this approach helps improve the reliability and specificity of responses while reducing the likelihood of off-topic or irrelevant content.

However, it's important to balance directional elements with enough flexibility for the model to leverage its capabilities. Over-constraining the prompt with too many directional stimuli can lead to rigid or artificial-sounding outputs.

When implementing DSP, focus on:

• Using clear, unambiguous directional cues

• Maintaining a natural flow despite the embedded stimuli

• Balancing guidance with creative freedom

• Testing different combinations of directional elements to find optimal results

By thoughtfully incorporating directional stimuli into your prompts, you can achieve more predictable and targeted outputs while maintaining the natural language capabilities of the model.

ReAct: Combining Reasoning and Action

ReAct (Reasoning and Acting) combines verbal reasoning with action generation in language models, enabling them to think through complex problems and take specific actions.

How ReAct Works

The ReAct framework operates by prompting language models to generate two key components:

• Verbal reasoning traces that show the model's thought process

• Specific actions based on that reasoning

This dual approach enables dynamic reasoning and high-level planning while allowing interaction with external environments. For example, when solving a complex question, ReAct will first reason about the necessary steps, then take actions to gather information, and finally synthesize the answer.

Performance and Benefits

ReAct has demonstrated significant improvements across various benchmarks:

• 34% performance improvement on ALFWorld for text-based game navigation

• 10% improvement on WebShop for web page navigation tasks

• Superior performance on question answering (HotPotQA) and fact verification (Fever) tasks

The framework offers several key advantages:

• Reduces hallucination by grounding reasoning in specific actions

• Prevents error propagation in chain-of-thought reasoning

• Provides interpretable decision-making processes

• Allows human inspection and correction during task execution

• Maintains robustness across diverse tasks

Applications

You should consider using ReAct when your application requires:

• Complex question answering that needs multiple steps

• Fact verification tasks requiring evidence gathering

• Navigation of text-based environments or web interfaces

• Tasks requiring both reasoning and interaction with external tools

• Scenarios where you need to inspect or control the model's behavior

ReAct's design makes it particularly effective for tasks that combine reasoning with real-world interactions, making it a valuable tool for building more capable and reliable AI systems. The framework's ability to generate both reasoning traces and actions makes it especially useful in production environments where transparency and control are crucial.

Reflexion

Reflexion implements a feedback loop mechanism, allowing language models to learn from their own outputs and improve through iteration by maintaining an episodic memory buffer of self-reflections.

How Reflexion Works

The Reflexion framework operates through a three-step cycle:

1. The model attempts to solve the given task

2. It generates verbal self-reflections about its performance

3. These reflections are stored in memory and used to inform future attempts

What makes Reflexion particularly versatile is its ability to incorporate various types of feedback signals. These can range from scalar values (like rewards or punishments) to free-form language feedback, and can come from either external sources (humans or other agents) or be internally generated by the model itself.

Performance Benefits

The effectiveness of Reflexion has been demonstrated across multiple domains:

• In decision-making tasks (AlfWorld), implementations showed a 22% improvement over 12 iterative learning steps

• For reasoning questions (HotPotQA), accuracy increased by 20%

• Most impressively, in Python programming tasks (HumanEval), Reflexion achieved a 91% pass@1 accuracy, significantly outperforming GPT-4's 80% baseline

Implementation Example

Here's how you might structure a basic Reflexion prompt:

Task: [Your specific task]
Previous Attempt: [Model's last solution]
Reflection: Let's analyze the previous attempt:
1. What worked well?
2. What could be improved?
3. What should we do differently?

New Solution:
[Model generates improved solution based on reflection]

The key to successful Reflexion implementation is maintaining a clear record of previous attempts and their corresponding reflections. This allows the model to build upon its experiences and avoid repeating past mistakes, creating a continuous improvement cycle that leads to increasingly refined outputs.

The technique is particularly effective for complex tasks requiring multiple iterations or when initial solutions might be suboptimal. By incorporating feedback and self-reflection, Reflexion provides a structured way for language models to refine their responses and achieve better outcomes through systematic improvement.

Multimodal Chain-of-Thought (CoT)

Multimodal Chain-of-Thought (CoT) extends traditional CoT prompting to handle tasks that combine text with other media types, enabling AI systems to reason about relationships between different modalities.

Understanding Image-Text Coherence

When implementing multimodal CoT, you need to consider different types of coherence relations between images and text. According to recent research, these relations can be categorized into:

• Visible relations: Direct descriptions of image content

• Action relations: Descriptions of events or actions shown

• Subjective relations: Evaluations or reactions to content

• Story relations: Background context or narrative

• Meta relations: Technical or contextual information about the image

Understanding these relations is crucial because they form the foundation of how your prompts should guide the model's reasoning process across modalities.

Benefits and Performance Impact

Multimodal CoT builds on the impressive performance gains seen in traditional CoT implementations. For example, research shows that in large models like PaLM 540B, CoT prompting improves performance by:

• 24% on mathematical reasoning tasks (SVAMP)

• 35% on symbolic reasoning tasks

• 19% on complex word problems (GSM8K)

When applied to multimodal tasks, these benefits extend to:

• More accurate image-text relationship understanding

• Better contextual reasoning about visual elements

• Improved ability to generate coherent explanations about visual content

Implementation Considerations

To effectively implement multimodal CoT, you should:

1. Use models of sufficient size (>100B parameters) as smaller models may produce illogical chains

2. Structure your prompts to explicitly address different coherence relations

3. Include reasoning steps that bridge visual and textual elements

4. Encourage the model to articulate its observations and conclusions about visual content

Remember that multimodal CoT requires careful prompt design to maintain coherence across modalities. Your prompts should guide the model to explain its reasoning about visual elements while maintaining logical connections to any textual context or requirements.

Graph Prompting

Graph prompting leverages knowledge graphs to enhance the capabilities of large language models, making it valuable for handling complex, interconnected information or structured reasoning.

When to Use Graph Prompting

You should consider graph prompting when your tasks involve:

• Complex relational reasoning between different pieces of information

• Need for comprehensive understanding across large datasets

• Requirements for connecting disparate information sources

• Structured knowledge representation and querying

Graph prompting excels in scenarios where traditional prompting methods might miss important connections or fail to capture the full context of related information. By incorporating knowledge graphs into your prompting strategy, you can enable the model to navigate through interconnected concepts and relationships more effectively.

Implementation and Benefits

The implementation of graph prompting typically combines LLM-generated knowledge graphs with graph machine learning techniques. This approach offers several key advantages:

• Enhanced context understanding: The model can better understand relationships between different pieces of information by visualizing them as connected nodes in a graph.

• Improved answer generation: By leveraging the graph structure, responses can be more comprehensive and diverse, especially for complex queries.

• Better information synthesis: The ability to connect multiple data points helps in generating more insightful and holistic answers.

Microsoft Research has demonstrated the effectiveness of this approach through GraphRAG, which extends Retrieval Augmented Generation (RAG) with knowledge graph capabilities. This implementation has shown particular success with complex datasets like the Violent Incident Information from News Articles (VIINA), where understanding relationships between different events and entities is crucial.

Advanced Applications

Graph prompting can be particularly powerful when:

• Working with both structured and unstructured data simultaneously

• Requiring improved ranking and relevance in information retrieval

• Needing to maintain consistency across related pieces of information

• Handling complex query chains that involve multiple steps of reasoning

One of the most significant advantages is the ability to perform text-to-query generation while maintaining the context of the broader knowledge structure. This enables more accurate and contextually relevant responses, especially in domains where relationships between different pieces of information are crucial for understanding.

By incorporating graph-based structures into your prompting strategy, you can create more sophisticated and capable AI applications that better handle complex, interconnected information while maintaining contextual accuracy and relevance.

What we’ve achieved in just ~7 months since inception is remarkable. $3M+ in ARR. Profitable. Grew team to 15. Zero funding!

Here’s the backstory:

At the beginning of the year, I took a sabbatical to immerse myself in the world of large language models (LLMs). I dove deep into retrieval-augmented generation (RAG), prompting strategies, fine-tuning techniques, and more.

Fast forward to mid-September, I got introduced to Marcel Santilli. Within minutes, it was clear—he was onto something big and had the experience + ambition to deliver on it.

Marcel's pitch:

• Brands need to become publishers.

• Founders need to become influencers.

• You need strategy + experts to deliver it, but AI can make it scalable.

Having previously helped with marketing before, I immediately got it—and Marcel had already proven the model’s effectiveness. The technical and product needs for the business were exactly the kinds of things I had spent the last year+ studying.

In the past four months, we’ve spent long hours validating a roadmap, getting the right team structure, and defining the GTM. Now we have the ambitious goal of reaching $15 million in ARR by 2025. To achieve this, my top priorities are building the right infra and putting together a great technical team.

We just opened two roles for founding engineers: https://growthx.ai/careers. We are a remote-first company hiring from anywhere, as long as we can have a 4-hour timezone overlap with San Francisco. If you’re interested, don’t hesitate to reach out!

We all know that LLMs struggle with writing unless you put in effort to improve their output. To save our eyeballs when reading AI-generated content, add this to every writing request you make to an LLM:

## Style guideline:

Avoid overused buzzwords (like ‘leverage,’ ‘harness,’ ‘elevate,’ ‘ignite,’ ‘empower,’ ‘cutting-edge,’ ‘unleash,’ ‘revolutionize,’ ‘innovate,’ ‘dynamic,’ ‘transformative power’), filler phrases (such as ‘in conclusion,’ ‘it’s important to note,’ ‘as previously mentioned,’ ‘ultimately,’ ‘to summarize,’ ‘what’s more,’ ‘now,’ ‘until recently’), clichés (like ‘game changer,’ ‘push the boundaries,’ ‘the possibilities are endless,’ ‘only time will tell,’ ‘mind-boggling figure,’ ‘breaking barriers,’ ‘unlock the potential,’ ‘remarkable breakthrough’), and flowery language (including ‘tapestry,’ ‘whispering,’ ‘labyrinth,’ ‘oasis,’ ‘metamorphosis,’ ‘enigma,’ ‘gossamer,’ ‘treasure trove,’ ‘labyrinthine’). Also, limit the use of redundant connectives and fillers like ‘moreover,’ ‘furthermore,’ ‘additionally,’ ‘however,’ ‘therefore,’ ‘consequently,’ ‘importantly,’ ‘notably,’ ‘as well as,’ ‘despite,’ ‘essentially,’ and avoid starting sentences with phrases like ‘Firstly,’ ‘Moreover,’ ‘In today’s digital era,’ ‘In the world of’. Focus on delivering the information in a concise and natural tone without unnecessary embellishments, jargon, or redundant phrases.

Summary

The main idea is that every single company that doesn't rely on cheating (corruption, government ties, etc.) to come to life and achieve long-term success (measured in profit/revenue) has to have at least one or more of these "powers":

1. Scale Economies

2. Network effect

3. Counter positioning

4. Branding

5. Switching costs

6. Cornered resources

7. Process

Power progression

The idea is that these powers aren't constant either, and especially in tech, things are much more dynamic than static. Just because you succeed with one of these powers doesn't mean it will sustain forever, and each power has different defensibility in the face of competition as well. He qualifies the impact and the moment you can rely on and try to chase each power based on the phases of the business:

1. Origination

2. Take-off phase

3. Stability phase

Each phase is self-explanatory, but his idea is that chasing specific barriers in the right phase of the business will build much higher competitive advantage and barriers for new players. Also, he is explicitly focused on business phases, not product life-cycle.

Powers applicable to Origination Phase

Phase where you are between the creation of the business, building everything and just launched and/or haven't achieved growth or market fit yet.

• Counter positioning

• Cornered resources

Applicable to Take-off Phase

Take-off phase is when the business starts to achieve fast growth.

• Scale economies

• Switching Costs

• Network effect

Applicable to Stability phase

The break between take-off and stability is when unit growth falls below about 30%-40% year.

• Branding

• Process power

Now to the description and examples of each power (in order of business phases):

Counter positioning (origination)

Can be identified by:

• An upstart who developed a superior business model.

• That business model's ability to successfully challenge well-entrenched incumbents.

• Steady accumulation of customers, all while incumbent remains paralyzed and unable to respond.

Ex: The new business model is superior to the incumbent's model due to lower costs and/or the ability to charge higher prices.

Barrier of entrance:

Incumbent(s) will eventually ask: "Am I better off staying the course, or adopting the new model?" If the answer is "stay off", then the barrier is high. A newcomer adopts a new, superior business model which the incumbent does not mimic due to anticipated damage to their existing business.

Counter-Positioning versus Disruptive Technologies (Clayton Christensen)

• Digital photography vs Kodak - this is a DT, but not CP.

• In-N-Out vs. McDonald's - this is CP, but not DT

• Netflix streaming vs. HBO via cable - this is both CP and DT

The case for playing humble: Cognitive Bias can play a role in deterring the incumbent. But the challenger, by its posture, may be able to influence such a move. How to attempt this? In its ascendancy, the challenger should avoid the temptation of trumpeting its superiority, instead suppressing that urge and adopting a tone of respect toward the incumbent.

Counter positioning is not exclusive. Different from other powers like Network Effect, strong counter positioning doesn't require (or create) a winner-takes-all situation.

Five stages of the incumbent when dealing with successful counter positioning from a new challenger:

1. Denial (aka, "we are not competitors")

2. Ridicule (aka, "we are better")

3. Fear

4. Anger

5. Capitulation (frequently too late)

Blockbuster execs showed all 5 stages in their public statements when facing Netflix. I personally dealt with founders on 1 and 2 many times (and always end up on 5 later).

Once market erosion becomes severe, a Counter-Positioned incumbent comes under tremendous pressure to do something; at the same time, they face great pressure to not upset the apple cart of the legacy business model. A frequent outcome of this duality? Let's call it dabbling: the incumbent puts a toe in the water, somehow, but refuses to commit in a way that meaningfully answers the challenge.

CP often underlies situations in which the following happens at the same time:

For the challenger:

• Rapid share gains

• Strong profitability (or promise of it)

For the incumbent:

• Share loss

• Inability to counter the entrant's moves

• Eventually management shake-up(s)

• Capitulation

Cornered resources (origination)

Benefit:

Cornered Resource can emerge in varied forms, offering uniquely different benefits. It might, for example, be preferential access to a valuable patent, such as that for a blockbuster drug; a required input, such as a cement producer's ownership of a nearby limestone source, or a cost-saving production manufacturing approach.

Ex: In the Pixar case, the Brain Trust produced an uncommonly appealing product—"superior deliverables"—driving demand with very attractive price/volume combinations in the form of huge box office returns.

Barrier:

Barrier in Cornered Resource is unlike anything we have encountered before. You might wonder: "Why does Pixar retain the Brain Trust?" Any one of this group would be highly sought after by other animated film companies, and yet over this period, and no doubt into the future, they have stayed with Pixar. In the case of spin casting technology, it is patent law, and in the case of cement inputs, it is property rights.

The Five Tests to validate Cornered Resource:

1. Individual/Exclusive: If a firm repeatedly acquires coveted assets at attractive terms, then the proper strategy question is, "Why are they able to do this?" Ex: if Exxon was able to persistently gain the rights to desirable properties, then understanding their path to that is key. Do relative scale allows them to develop better discovery processes? If so, their discovery processes are the Cornered Resource.

2. Non-arbitraged: cost to keep the resource should be low enough to afford consistent differential returns (ex: a movie hiring Brad Pitt might move the box office but won't necessarily 10x the cost).

3. Transferable: If a resource creates value at a single company but would fail to do so at other companies, then isolating that resource as the source of Power would entail overlooking some other essential complement beyond operational excellence.

4. Ongoing: In searching for Power, a strategist tries to isolate a causal factor that explains continued differential returns. Ex. Post-it notes patent lasted for 30 years

5. Sufficient: The final Cornered Resource test concerns completeness: for a resource to qualify as Power, it must be sufficient for continued differential returns, assuming operational excellence. Ex. George Fisher joining Kodak wasn't enough.

Switching costs (take-off phase)

Ex: SAP - 43% of users are unhappy but 85% say they will continue to pay for the product, and the sales validate their statements.

Benefit:

A company that has embedded Switching Costs for its current customers can charge higher prices than competitors for equivalent products or services. Benefit only possible when you have additional offerings to sell.

Barrier:

To offer an equivalent product, competitors must compensate customers for Switching Costs. The firm that has previously roped in the customer, then, can set or adjust prices in a way that puts their potential rival at a cost disadvantage, rendering such a challenge distinctly unattractive.

Types of switching costs:

• Financial: $ cost of investing and purchasing new solutions (ex: SAP)

• Procedural: loss of familiarity or risk of adoption of a new product (ex: "nobody was ever fired for choosing IBM")

• Relational: Affection with the provider (ex: sales team) or product/identity as a user.

Network Effect (take-off phase)

Benefit:

Network Economies occur when the value of a product to a customer is increased by the use of the product by others (Branch Out vs LinkedIn)

A company in a leadership position with Network Economies can charge higher prices than its competitors, because of the higher value as a result of more users.

Barrier:

For Network Economies, the barrier is the unattractive cost/benefit of gaining share, and this can be extremely high. In particular, the value deficit of a follower can be so large that the price discount needed to offset this is unthinkable.

Example: The value of LinkedIn's HR Solutions Suite comes from the numbers of LinkedIn users, so LinkedIn can charge more than a competing product with fewer participants.

Characteristics:

Industries exhibiting Network Economies often exhibit these attributes:

1. Winner take all: Businesses with strong Network Economies are frequently characterized by a tipping point: once a single firm achieves a certain degree of leadership, then the other firms just throw in the towel. Game over—the P&L of a challenge would just be too ugly. For example, even a company as competent and with as deep pockets as Google could not unseat Facebook with Google+.

2. Boundedness: As powerful as this Barrier is, it is bounded by the character of the network, something well-demonstrated by the continued success of both Facebook and LinkedIn. Facebook has powerful Network Economies itself but these have to do with personal not professional interactions. The boundaries of the network effects determine the boundaries of the business.

3. Decisive early product: Due to tipping point dynamics, early relative scaling is critical in developing Power. Who scales the fastest is often determined by who gets the product most right early on. Facebook's trumping of MySpace is a good example.

Scale Economies (take-off)

The quality of declining unit costs with increased business size is referred to as Scale Economies.

Benefit:

Some condition which yields material improvement in the cash flow of the Power wielder via reduced cost, enhanced pricing and/or decreased investment requirements.

Barrier:

Some obstacle which engenders in competitors an inability and/or unwillingness to engage in behaviors that might, over time, arbitrage out this benefit.

For Scale Economies, the Benefit is straightforward: lowered costs. In the case of Netflix, their lead in subscribers translated directly into lower content costs per subscriber for originals and exclusives.

Example:

Netflix paid $100M for House of Cards and their streaming business had 30M customers, then the cost per customer was three dollars and change. In this scenario, a competitor with only one million subscribers would have to ante up $100 per subscriber.

This situation creates a very difficult position for Netflix's smaller-scale streaming competitors. If they offer the same deliverable as Netflix, similar amounts of content for the same price, their P&L will suffer. If they try to remediate this by offering less content or raising prices, customers will abandon their service and they will lose market share. Such a competitive cul-de-sac is the hallmark of Power.

Branding (stability phase)

Branding as power to charge higher prices and sustain market share.

Ex: Purchased a Diamond ring at Tiffany for $16,600 and one of similar size and cut at Costco for $6,600, then asked a reputable gemologist and appraiser to assess the rings' values: Costco ring at $8,000 plus setting costs, more than $2,000 above the selling price. Assessed Tiffany ring at $10,500 plus setting costs at a non-brand-name retailer.

"You got exactly what they said you were getting. Anything that is brand-name and has developed a reputation that Tiffany has developed, they've earned it over the years for quality control. You can go there [and] you don't have to think twice about your purchase. And you pay for that."

Benefit:

A business with Branding power is able to charge a higher price for its offering due to one or both of these two reasons:

1. Affective valence - The built-up associations with the brand elicit good feelings about the offering, distinct from the objective value of the good. (Ex Tiffany's)

2. Uncertainty reduction - A customer attains "peace of mind" knowing that the branded product will be just as expected. Consider another example: Bayer aspirin. Search for aspirin on Amazon.com and you will see a 200 count of Bayer 325 mg. aspirin for $9.47 side-by-side with a 500 count of Kirkland 325 mg. aspirin for $10.93. So Bayer has a price per tablet premium of 117%. Some customers still would prefer the Bayer because of diminished uncertainty: Bayer's long history of consistency makes customers more confident that they are getting exactly what they want.

Note that the Benefit from Branding does not depend on prior ownership, as with Switching Costs.

Barrier:

A strong brand can only be created over a lengthy period of reinforcing actions (hysteresis), which itself serves as the key Barrier.

Process (stability phase)

Ex. Toyota Process (TPS)

Benefit:

A company with Process Power is able to improve product attributes and/or lower costs as a result of process improvements embedded within the organization. For example, Toyota has maintained the quality increases and cost reductions of the TPS over a span of decades; these assets do not disappear as new workers are brought in and older workers retire.

Barrier:

These process advances are difficult to replicate, and can only be achieved over a long time period of sustained evolutionary advance. This inherent speed limit in achieving the Benefit results from two factors:

1. Complexity - Returning to our example: automobile production, combined with all the logistic chains which support it, entails enormous complexity. If process improvements touch many parts of these chains, as they did with Toyota, then achieving them quickly will prove challenging, if not impossible.

2. Opacity - The development of TPS should tip us off to the long time constant inevitably faced by would-be imitators. The system was fashioned from the bottom up, over decades of trial and error. The fundamental tenets were never formally codified, and much of the organizational knowledge remained tacit, rather than explicit. It would not be an exaggeration to say that even Toyota did not have a full, top-down understanding of what they had created —it took fully fifteen years, for instance, before they were able to transfer TPS to their suppliers. GM's experience with NUMMI also implies the tacit character of this knowledge: even when Toyota wanted to illuminate their work processes, they could not entirely do so.

Innovation is the key to achieving any of the powers

Innovation is achieved by creating compelling value. There are three distinct paths to creating compelling value.

1. Capabilities-Led Compelling Value: Adobe Acrobat. Here the key capability brought to bear was Adobe's existing fluency at the intersection of software and graphics.

2. Customer-Led Compelling Value: Corning Fiber Optics. Interaction at a distance. The uncertainty in this case is technical: "Can we invent it?"

3. Competitor-Led Compelling Value: the Sony PlayStation. Sony perceived the value in immersion of 3D graphics and the gap in the competitors' offerings (Sega/Nintendo) to double down on 3D and launch PS1.

I've been absent lately, as some of you noticed. I'd set aside three months for a focused "studying" period, and it was a lot of fun to dive deep into prompt engineering, evals, LLM product lifecycles, agents, and more. However, transitioning from my previous full-time job to 8+ hour study days was challenging. Because I'm usually not great at taking breaks (I think I never took more than 2 weeks off), this time I was determined to stick to my three-month deadline and disconnect. I hit the deadline and went completely offline. I had a blast spending a month road cycling in Spain & Germany and a lot of time with family.

Now, I'm recharged and excited to be back!

Here are some learnings, a new project I shipped this week, and some plans for the next few months:

Aimap.today: An Experiment with Agents & Flows

I recently focused on learning about agent tooling to build an AI company researcher for personal use.

The plan: Create an agent to find funding news, identify AI-related companies, analyze their websites, build a knowledge base, write a long form article about it, and categorize them into three tiers (foundational AI tech, AI-enabled, and AI users).

I explored several agent frameworks: CrewAI, LangGraph (LangChain’s agent take), LlamaIndex's Llama-Agents (a microservice take on agents, where each agent is it’s own service), and AgentOps (very cool instrumentation tool). I also tried building flows with AirOps and n8n. In the process of fetching and enriching the data I also tried Jina Reader/SERP, You.com API, Apollo API, and Clay agent.

Ultimately, for now, I decided to take a stab at coding my own thing. Most existing frameworks focus on LLMs talking to each other, but all I needed was a workflow/pipeline to chain multiple processes together.

My current boilerplate is in Ruby, and I wasn't satisfied with the existing packages for API clients like Claude and OpenAI. So, I coded these from scratch as well. The whole project, including study time, took about two weeks.

👉 The result is: https://aimap.today . Please, check it out and let me know what you think.

I built this for myself and plan to add new companies weekly. There's more I want to do, including a weekly digest for newly researched companies. If you sign-up you’ll be notified when I finish the weekly digest part.

My next step is to explore more out-of-the-box platforms like AirOps and n8n, and try to covert my little workflow to these tools.

Code-wise the pipeline currently looks like this:

Joined Techstars for the First San Francisco class

I'm thrilled to share that I’ll be helping Techstars’ first San Francisco class as an EIR (Entrepreneur in Residence). I've been following my friend and former boss, Neal, as he ran Techstars Chicago and Oakland. When he mentioned the new San Francisco class, I jumped at the chance to participate.

The cohort is impressive, featuring many interesting AI-enabled businesses. I'm excited to work with all the founders. You can view the full cohort here: https://www.techstars.com/newsroom/techstars-announces-inaugural-class-of-san-francisco-accelerator

It's great to be back. Building aimap.today has been an a cool learning experience, and I'm eager to continue improving and expanding it.

Joining Techstars as an EIR is another exciting opportunity I'm looking forward to.

I also may be joining a team on a new project soon, but I'll save that for future updates.

Going forward, my goal is to resume weekly updates.

I'm starting to plan what to do next in the long run, and I’m entertaining the idea of joining a team. I think things might be moving too fast and too much demand for skills for risking time on solo projects.

Three types of play

Started considering which spaces are interesting now. I think this take from Hamilton Helmer is a good one: Three types of play:

1. The tech play: example of semiconductors, you get Intel and etc.

2. The first-tier play: companies that wouldn't exist before semiconductors (Microsoft)

3. The usage play: car companies with tons of semiconductors. They already existed, just got better.

The whole video is worth watching, but heres’ the time-stamp about AI.

I'm personally more interested in the second play. And finding these companies early and identifying which ones are already in a good direction is so hard.

Brainstorm exercise

A good place to start could be looking at common roles and how they will be impacted and can be augmented or even replaced.

A programmer's job is already seriously impacted, customer support even more so. Same for SDRs. Some other roles, like localization, have already been pretty much replaced. These highly impacted roles are the things I'm most interested in.

So, just as a thought exercise toying with Python + Langchain, I wrote a 4-step chain with Claude 3.5 to do following:

1. Brainstorm departments and roles common across any vertical

2. List the most common tasks in those roles

3. Brainstorm ideas for how each could be augmented or automated

4. (Rank them by difficulty and confidence, but this was just for nudging the model)

I only spent a couple of hours and $3 in credits, but the results turned out better than I was expecting.

There's obviously some nonsense and some things that are regular ML models more than LLMs in the output (my fault, I should have optimized the last step of the chain better), but a lot of solid ones that I think haven't happened yet, and some others in the “augmentation” category fully in progress like Figma, Intercom, Cursor, GitHub Copilot, etc. If you squint and look at the roles with the best ideas, you can see Piper, Fin, Harvey, Wisq, Devin, 11x.ai, etc

You can see the whole list here:

https://docs.google.com/spreadsheets/d/1vpXzfUFNhIZJp9CEmiZqGKaVrP_Xx8McMZZlLCxiHNg/edit?gid=0#gid=0

Next step, I guess, is to start collecting companies and maybe have a little agent help me with the research.

—

I haven't had time to think through the whole list yet, but here...Some of the ones that stood out that the ideas are mostly very doable, or already happening:

Traditional machine learning

When you move beyond simply calling APIs, you'll encounter the traditional ML stack and terminology. For instance, if you need to adjust your embeddings and choose a different model, you'll need a solid grasp of metrics like BPB, BPC, Perplexity, and so on. The same goes for evaluating production data.

I also prefer not to have components in a stack that I don't understand at a high level, even if I don't have to work with them directly (for example, if I'm an EM on the product side or a product manager).

For ML theory I’m currently half way through this course. It may be a few years old, but I appreciate how practical and real-world focused it is. If anyone has other suggestions, I'm very open to them.

I can't recall the exact source of this slide (I think it was from Hamel Hussain’s talk on AI Engineer conf this month) , but I believe it provides a reasonable overview - and my goal is to have at least a basic understanding of everything under the horizontal line.

Evaluations

Last week, I began studying evaluations more seriously for two reasons. First, I'm working on understanding the various stages of the development cycle. Second, I need to establish a better workflow for iterating on our AI bot at Canopy.

For theory around evals I really liked Chip’s third chapter of her WIP book. This post about Evaluations from Hamel Husain is also so good, and this one as well.

This is would be the ideal cycle for me:

At Canopy, our tech stack is entirely Ruby, and our AI Assistant was built using direct API calls to OpenAI. While I did implement some observability, I still need a more robust way to monitor the quality of answers in production and a solid test suite that I can run when updating the model or modifying the prompts.

Ideally, I would have automated tests similar to regular test units that I can run whenever I'm iterating on prompts, models, or content format for the rag. Once things are deployed, I'd like to have comparable quality metrics for the production data, with alerts if quality drops below certain thresholds and proper visibility.

There are quite a few LLM-focused evaluation tools available, and many of them are really solid. I don't mean to dismiss their work, but for my specific context, here's my current assessment:

1. They do too much

   1. Most tools attempt to offer solutions for all four stages of my diagram, but these companies are often small, seed-level teams building in a rapidly evolving space. Many of these products feel either half-baked or are still trying to figure out their business model. I'd prefer a tool that focuses solely on the two evaluation parts (dev & production) and doesn't try to handle the prototyping and deployment aspects.

2. Their business models are still unclear

   1. Many tools are in the typical early-stage B2B mode, lacking a product-led path and aiming to lock users into annual contracts with case-by-case pricing. I understand this is important in a new market with early-stage products, but I'd love to see more product-led options (especially for Canopy's context, a small team where the bot isn't their core feature yet).

3. Vendor lock in / dependency risk

   1. To enable tracing, many tools heavily wrap around API clients for services like OpenAI, and some don't even offer public API versions. Tools that offer "solutions" for the deployment part often proxy calls under their own API, so instead of calling OpenAI directly, you call their API. This introduces a third-party proxy as a potential failure point, in addition to OpenAI/Anthropic/etc., and it's built by an understaffed seed company.

4. Pretty much no solutions for non-Python/JavaScript stacks

   1. Most niceties for automatic logging/tracing and basic metrics like latency come from their SDK, which are often limited to Python/JS. While understandable given the early stage, most tools also don't offer a public API. For more elaborate setups, a reasonable workflow could be to follow the traditional ML path of using Python for the stack and wrapping it as an API for other languages to call, instead of relying on a proxy maintained by a tiny company in a crowded space. It's a bit of a bummer because for AI products that only need LLMs, it's an overcomplication in the stack. You'll definitely need query augmentation, retries, rate-limiting controls, and other things a product-level software already handles, and now you'll need to duplicate that in the Python stack too. It's one thing when you're making an API around a model you control, but it's another when you're making an API around another API.

If you're already using a Python or JS stack, you have fewer things to worry about. However, for Canopy (Ruby stack), the situation is a bit different. I think I'll go with the following approach:

• First: Find a tool that has a public API support for sending run data for production, build a small Ruby client for that, and do production level monitoring only (similar to how’d use a service like Sentry). Ignore all the other 3 stages of the Experiment and Deploy part of these tools.

  • Second:

    • Have a good unit test coverage for the direct calls to OpenAI with VCR and reimplement some of the basic eval metrics that relevant to my context like Answer Relevance, Context Recall, or Context Relevancy.

    • Or use something like Promptfoo, Ragas or GuardRails for the local unit tests.

I haven't decided which product to use yet, but reading through their documentation, examining the code of their SDKs, and exploring their offerings has been a good learning process already.

In no particular order, here's the current list of evaluation products/projects that I'm aware of (although I haven't had a chance to thoroughly check all of them yet):

• https://mlflow.org

• https://www.vellum.ai/

• https://athina.ai

• https://www.patronus.ai

• https://wandb.ai/site/weave

• https://autochain.forethought.ai/

• https://ragas.io/

• https://github.com/agenta-ai/agenta

• https://klu.ai/

• https://www.braintrustdata.com/

• https://humanloop.com/

• https://www.parea.ai/

• https://docs.llamaindex.ai/en/stable/understanding/evaluating/evaluating/

• https://www.langchain.com/langsmith

• https://github.com/langfuse/langfuse

• https://www.rungalileo.io/

• https://www.guardrailsai.com/

• https://deepchecks.com

• https://arize.com

• https://github.com/UKGovernmentBEIS/inspect_ai

If you're using a tool that you really like or if you have any feedback or a different perspective on the market, I'd love to hear your thoughts.

While I haven't done much hands-on AI engineering this last two weeks, I've been keeping up with theory. Here's some of the notes:

Daily Prompting Paper

For a while now, I've been checking Arxiv daily for new papers on Prompt Engineering. It's a topic that's usually easy to understand, even in academic papers, so great way to kill time between tasks.

Many poor results in AI come from inadequate prompting skills + techniques like prompt chaining and retries, weren't practical before, but now make sense.

In June, this excellent paper was published. I spent the last couple of weeks reading it thoroughly, along with many of its citations. I've compiled some notes here, and I'm also planning to create a better structured resource with more examples and additional notes I didn't include.

Meetups and Conferences

I've been attending several meetups and conferences, including AIQCon yesterday. It's interesting to see the growing focus on AI Engineering (meaning implementing GenAI in products). Last year, there a lot of interest in LLMs themselves, but now the emphasis is on making LLMs perform well in real-world applications.

Key topics include:

• RAG (Retrieval-Augmented Generation): Techniques to improving embeddings, content augmentation, and advanced techniques like GraphRAG.

• Evaluation: Setting up good test suites, deciding what to test, and determining who should write the tests.

• No fine-tuning: The consensus is to focus on prompting, RAG, and evaluation before considering fine-tuning. Lot of pain and scar from fine-tuning already.

While some events feel like they have a high noise-to-signal ratio, I always learn a lot as a newcomer. Also, it’s such a great time to live in SF. For future conferences, I plan to research the participants and their topics more thoroughly beforehand.

A useful resource from yesterday's conference: https://huyenchip.com/llama-police.html by Chip Huyen (well worth following on X).

Books

On the topic of learning more theory, I'm currently reading two books:

1. Programming Machine Learning

https://www.amazon.com/Programming-Machine-Learning-Zero-Deep/dp/1680506609

I first read this when it was released, but I've forgotten most of it. Although it's a bit outdated and pre-LLM, I like the author's writting. Given how close AI Engineering is to traditional ML, I want to revisit this material.

2. Neural Nets mini-course by 3Blue1Brown

Another basic entry level material I was revisiting this week is this mini-course on neural nets: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

2. Oreilly’s AI Engineering book

https://www.oreilly.com/library/view/ai-engineering/9781098166298/cover.html

This book was recommended by a close person, and I attended a talk by the author yesterday - it was one of my favorites, despite being too short. Only three chapters are available now, so I'll keep an eye out for future updates.

I read "The Prompt Report: A Systematic Survey of Prompting Techniques" from start to finish, as well as many of the papers it cited and extracted all the things that I thought were useful techniques to have in the text-prompting toolbelt (left out all the benchmarking and multi-modal techniques).

Here's everything; maybe you'll find it useful too:

1. In-Context Learning (ICL) and Few-Shot Prompting

In-Context Learning (ICL) is a type of few-shot prompting technique. It allows AI models to perform tasks based on a small number of examples or instructions within the prompt, without additional training.

1.1 Key Concepts of ICL

• Skills can be derived from exemplars or instructions in the prompt

• May utilize pre-existing knowledge rather than true "learning"

• Viewed as task specification rather than acquiring new skills

• Current research focuses on optimization and understanding of ICL

1.2 Examples of ICL

1. Learning from exemplars:

System:
Translate English to French:
English: Hello
French: Bonjour
English: Goodbye
French: Au revoir
---

User:
English: How are you?

2. Learning from instructions:

System Prompt:
Identify the subject and predicate in sentences. The subject is the doer, and the predicate is the action.

Example:
Sentence: The cat chased the mouse.
Subject: The cat
Predicate: chased the mouse

Now, your turn:
--

User:
Sentence: The sun rises in the east.
[Model identifies subject and predicate]

3. Task specification:

Prompt:
Generate a haiku about artificial intelligence. A haiku is a three-line poem with five syllables in the first line, seven in the second, and five in the third.

ICL enables flexible task performance without fine-tuning, but its effectiveness depends on prompt design and example selection. The technique is limited by the model's maximum input length and may not represent true learning of new skills.

1.3 Few-Shot Prompting Design Decisions

The effectiveness of ICL and few-shot prompting techniques depends significantly on how the examples (exemplars) are designed and presented. Here are six critical design decisions:

1.3.1 Exemplar Quantity

• Generally, more exemplars improve performance, especially in larger models.

• Benefits may diminish beyond 20 exemplars in some cases.

1.3.2 Exemplar Ordering

• The order of exemplars affects model behavior.

• On some tasks, exemplar order can cause accuracy to vary from below 50% to over 90%.

1.3.3 Exemplar Label Distribution

• The distribution of labels in exemplars can bias the model's output.

• Example: 10 exemplars of one class and 2 of another may bias the model towards the first class.

1.3.4 Exemplar Label Quality

• The necessity of strictly valid demonstrations is unclear.

• Some studies suggest label accuracy may be irrelevant, while others show significant impact.

• Larger models often handle incorrect or unrelated labels better.

Note: When automatically constructing prompts from large datasets, consider studying how label quality affects your results.

1.3.5 Exemplar Format

• Common format: "Q: {input}, A: {label}"

• Optimal format may vary across tasks; experiment with multiple formats.

• Formats common in the training data may lead to better performance.

1.3.6 Exemplar Similarity

• Selecting exemplars similar to the test sample is generally beneficial.

• In some cases, more diverse exemplars can improve performance.

These design decisions critically influence output quality in few-shot prompting and ICL.

2. Few-Shot Prompting Techniques

Few-Shot Prompting can be challenging to implement effectively. Here are some techniques that can be used in a supervised setting:

2.1 Techniques for Selecting Exemplars

2.1.1 K-Nearest Neighbor (KNN)

• Run your examples through KNN and selects exemplars similar to test to boost performance

• Effective but can be time and resource-intensive during prompt generation

2.1.2 Vote-K

• Two-stage process for selecting similar exemplars:

1. Model proposes useful unlabeled candidate exemplars for annotation

2. Labeled pool is used for Few-Shot Prompting

• Ensures diversity and representativeness of exemplars

2.1.3 Self-Generated In-Context Learning (SG-ICL)

• Uses a generative AI to automatically generate exemplars

• Better than zero-shot scenarios when training data is unavailable

• Generated samples are not as effective as actual data

To Consider: https://developer.nvidia.com/blog/leverage-our-latest-open-models-for-synthetic-data-generation-with-nvidia-nemotron-4-340b/

2.2 Prompt Mining

• Process of discovering optimal "middle words" in prompts through large corpus analysis

• Aims to find prompt formats that occur frequently in the corpus

• Frequently occurring formats may lead to improved prompt performance

• Example: Instead of using "Q: A:" format, finding a similar but more common format in the corpus

These techniques can help improve the effectiveness of few-shot prompting by optimizing exemplar selection and prompt format. The choice of technique may depend on the specific task, available resources, and the characteristics of the dataset and model being used.

3. Zero-Shot Prompting Techniques

Zero-Shot Prompting uses no exemplars, in contrast to Few-Shot Prompting. Here are several standalone zero-shot techniques:

3.1 Role Prompting (Persona Prompting)

• Assigns a specific role to the AI in the prompt (e.g., "You are a travel writer")

• Can improve outputs for open-ended tasks and sometimes improve accuracy on benchmarks

Example:

You are an experienced travel writer for a luxury lifestyle magazine. Describe the experience of visiting the {{city}} in {{country}}, focusing on the sensory details and exclusive experiences a high-end traveler might enjoy.

3.2 Style Prompting

• Specifies desired style, tone, or genre in the prompt

• Similar effect can be achieved using role prompting

Examples:

Write a short sales pitch for a {{product}} in a persuasive, benefit-focused style. Emphasize how the product solves customer problems in a straightforward way.

Compose a follow-up email to a potential client {{client_name}} after a sales call, using a friendly and conversational tone. The goal is to build rapport and schedule a product demo.

Convert the following git commit message and code diff into a clear, concise changelog entry. Use a professional tone and focus on the user-facing changes and improvements. The entry should be understandable to non-technical users.

3.3 Emotion Prompting

• Incorporates phrases of psychological relevance to humans (e.g., "This is important to my career")

• May lead to improved performance on benchmarks and open-ended text generation

3.4 System 2 Attention (S2A)

1. Asks the AI to rewrite the prompt, removing unrelated information

2. Passes the new prompt to the AI for a final response

Example:

System prompt:

Given the following text by a user, extract the part that is unbiased and not their opinion, so that using that text alone would be good context for providing an unbiased answer to the question portion of the text.

Please include the actual question or query that the user is asking. Separate this into two categories labeled with “Unbiased text context (includes all content except user’s bias):” and “Question/Query (does not include user bias/preference):”.
--

Quest: Text by User: [ORIGINAL INPUT PROMPT]

https://arxiv.org/pdf/2311.11829

3.5 SimToM (Simulation Theory of Mind)

• Deals with complicated questions involving multiple people or objects

1. Establishes the set of facts one person knows

2. Answers the question based only on those facts

• Helps eliminate the effect of irrelevant information in the prompt

Single Prompt:

Your task is in two steps.
Step 1. output only the events that
{character_name} knows about.
Step 2. Imagine you are {character_name},
then answer a question based only on the
events {character_name} knows about.
Story: {story}
Question: {question}

The following is a sequence of events about some characters, that takes place in multiple locations.
Your job is to output only the events that the specified character, {character}, knows about.
Here are a few rules:
1. A character knows about all events that they do.
2. If a character is in a certain room/location, that character knows about all other events that happens in the room. This includes other characters leaving or exiting the location, the locations of objects in that location, and whether somebody moves an object to another place. 3. If a character leaves a location, and is NOT in that location, they no longer know about any events that happen within that location. However, they can re-enter the location.
Story: {story}
What events does {character} know about?
Only output the events according to the
above rules, do not provide an
explanation.

{perspective}
You are {name}.

Based on the above information, answer the following question:

{question}

Keep your answer concise, one sentence is
enough. You must choose one of the above
choices.

Imagine you are {name}, and consider this story that has an unexpected event.

{story}

If the last sentence of the story says {name} notices, sees or realizes the unexpected event in this story, simply output the original story with nothing
changed.

However, if the sentence says you are not aware of the changes in this story, output only the events you know, i.e., the sentences before the unexpected event happens.

Output either the original story or the edited story, nothing else.

Format your answer as follows:
Sees/Notices/Realizes: (Yes/No)

Story:

Answer the questions based on the context.
Keep your answer concise, few words are
enough, maximum one sentence. Answer as

’Answer:<option>)<answer>’.
{perspective}

You are {name}.
{question}

Choose the most straightforward answer.

Task Instructions:

I will give you an excerpt. Your task involves three steps:

1. Identify the Sentence Describing the Unexpected Change:

Find the sentence that describes how the situation unexpectedly changed.

2. Determine the Main Character's Awareness:

Identify if the main character comes to know about, or notices, this change at the end.  

3. Edit Based on Awareness:

- If the main character does not know about this change, edit the excerpt and output the part of the excerpt BEFORE the sentence that describes the change.

- If the main character does know about the change, do not edit the excerpt, and output the original story.

Examples:  

Story:

Olumide, a skilled woodcarver in a Nigerian village, is preparing to carve a beautiful sculpture for the village chief. Olumide wants to use a sharp chisel to create intricate details on the sculpture. Olumide observes his set of chisels and sees one that appears to be sharp and in perfect condition. However, while Olumide is talking to a fellow artisan, a child from the village accidentally drops the chisel, causing it to become blunt and damaged. Olumide does not notice the damaged chisel on the ground.

Sentence: However, while Olumide is talking to a fellow artisan, a child from the village accidentally drops the chisel, causing it to become blunt and damaged.

Knows about or notices change: No  

Edit:
Olumide, a skilled woodcarver in a Nigerian village, is preparing to carve a beautiful sculpture for the village chief. Olumide wants to use a sharp chisel to create intricate details on the sculpture. Olumide observes his set of chisels and sees one that appears to be sharp and in perfect condition.

Story:

{story}

https://arxiv.org/pdf/2311.10227

3.6 Rephrase and Respond (RaR)

• Instructs the AI to rephrase and expand the question before generating the final answer

• Can be done in a single pass or as separate steps

• Has shown improvements on multiple benchmarks

https://arxiv.org/pdf/2311.04205

3.7 Re-reading (RE2)

• Adds the phrase "Read the question again:" to the prompt and repeats the question

• Simple technique that has shown improvement in reasoning benchmarks, especially with complex questions

Example:

########### Task Instruction ##############

# You will write python program to solve math problems.

# You will only write code blocks.

# Please generate your code block in 'def solution()' function, and thus it can be executed by python interpreter. You don't need to call 'solution()' function because

# it will be called by the system.

# The concrete format of 'solution()' is as follows:

# def solution():

# """<question>"""

# <your code>

# result = <your result>

# return result

#########################################

# Q: {question}

# Read the question again: {question}

# Your defined "solution()" function with comments here.

https://arxiv.org/pdf/2309.06275

3.8 Self-Ask

1. Prompts the AI to decide if follow-up questions are needed

2. If needed, the AI generates these questions

3. The AI answers the follow-up questions

4. Finally, the AI answers the original question

These zero-shot techniques can be used individually or in combination with other prompting methods to improve AI performance on various tasks without the need for task-specific examples.

Paper has both a zero-shot and the following four-shot example:

prompt = ['''Question: Who lived longer, Muhammad Ali or Alan Turing?
Are follow up questions needed here: Yes.
Follow up: How old was Muhammad Ali when he died?
Intermediate answer: Muhammad Ali was 74 years old when he died.
Follow up: How old was Alan Turing when he died?
Intermediate answer: Alan Turing was 41 years old when he died.
So the final answer is: Muhammad Ali 

Question: When was the founder of craigslist born?
Are follow up questions needed here: Yes.
Follow up: Who was the founder of craigslist?
Intermediate answer: Craigslist was founded by Craig Newmark.
Follow up: When was Craig Newmark born?
Intermediate answer: Craig Newmark was born on December 6, 1952.
So the final answer is: December 6, 1952

Question: Who was the maternal grandfather of George Washington?
Are follow up questions needed here: Yes.
Follow up: Who was the mother of George Washington?
Intermediate answer: The mother of George Washington was Mary Ball Washington.
Follow up: Who was the father of Mary Ball Washington?
Intermediate answer: The father of Mary Ball Washington was Joseph Ball.
So the final answer is: Joseph Ball 

Question: Are both the directors of Jaws and Casino Royale from the same country? 
Are follow up questions needed here: Yes. 
Follow up: Who is the director of Jaws? 
Intermediate Answer: The director of Jaws is Steven Spielberg. 
Follow up: Where is Steven Spielberg from? 
Intermediate Answer: The United States. 
Follow up: Who is the director of Casino Royale? 
Intermediate Answer: The director of Casino Royale is Martin Campbell. 
Follow up: Where is Martin Campbell from? 
Intermediate Answer: New Zealand. 
So the final answer is: No

Question: ''', 
'''
Are follow up questions needed here:''', ]

https://github.com/ofirpress/self-ask?tab=readme-ov-file

4. Thought Generation Techniques

Thought Generation encompasses techniques that prompt the AI to articulate its reasoning while solving a problem. These methods can significantly enhance performance in mathematics and reasoning tasks.

4.1 Chain-of-Thought (CoT) Prompting

• Uses few-shot prompting to encourage the AI to express its thought process

• Prompt includes an exemplar with a question, reasoning path, and correct answer

• Example format:

Q: Jack has two baskets, each containing three balls. How many balls does Jack have in total?

A: One basket contains 3 balls, so two baskets contain 3 * 2 = 6 balls.

Q: {QUESTION}

A:

4.2 Zero-Shot-CoT

• Requires no exemplars

• Appends a thought-inducing phrase to the prompt, such as:

• "Let's think step by step."

• "Let's work this out in a step by step way to be sure we have the right answer."

• "First, let's think about this logically."

• Task-agnostic and doesn't require exemplars

4.3 Step-Back Prompting

• Modification of CoT

• First asks a generic, high-level question about relevant concepts or facts

• Then proceeds with reasoning

• Has shown significant improvements on reasoning benchmarks

https://arxiv.org/pdf/2310.06117

4.4 Analogical Prompting

• Similar to Self-Generated In-Context Learning (SG-ICL)

• Automatically generates exemplars that include Chains of Thought

• Demonstrated improvements in mathematical reasoning and code generation tasks

https://arxiv.org/pdf/2310.01714

4.5 Thread-of-Thought (ThoT) Prompting

• Uses an improved thought inducer: "Walk me through this context in manageable parts step by step, summarizing and analyzing as we go."

• Particularly effective in question-answering and retrieval settings with large, complex contexts

4.6 Tabular Chain-of-Thought (Tab-CoT)

• Zero-Shot CoT prompt that makes the AI output reasoning as a markdown table

• Tabular design improves the structure and reasoning of the output

https://arxiv.org/pdf/2305.17812

These Thought Generation techniques aim to improve the AI's problem-solving capabilities by making its reasoning process more explicit and structured. They can be particularly useful for complex tasks that require step-by-step reasoning or analysis.

4.7 Few-Shot Chain-of-Thought (CoT) Techniques

Few-Shot CoT techniques present the AI with multiple exemplars that include chains-of-thought. This approach can significantly enhance performance and is sometimes referred to as Manual-CoT or Golden CoT.

4.7.1 Contrastive Chain-of-Thought

• Shows the AI how not to reason

• Has shown significant improvement in areas like Arithmetic Reasoning and Factual QA

https://arxiv.org/pdf/2311.09277

4.7.2 Uncertainty-Routed CoT Prompting

1. Samples multiple CoT reasoning paths

2. Demonstrates improvement on both GPT-4 and Gemini Ultra models

Note: Didn’t read in full

https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf

4.7.3 Complexity-based Prompting

• Two major modifications to CoT:

1. Selects complex examples for annotation and inclusion in the prompt

2. During inference, samples multiple reasoning chains and uses a majority vote among chains exceeding a certain length threshold

• Premise: Longer reasoning indicates higher answer quality

• Has shown improvements on math reasoning datasets

Note: GPT-3 / Codex and tested

https://arxiv.org/pdf/2210.00720

4.7.4 Active Prompting

1. Starts with some training questions/exemplars

2. Asks the AI to solve them

3. Calculates uncertainty (disagreement in this case)

4. Asks human annotators to rewrite the exemplars with highest uncertainty

Note: another math-focused, but this could likely work well with reasoning q too.

https://arxiv.org/pdf/2302.12246

4.7.5 Memory-of-Thought Prompting

• Using unlabeled training exemplars to build Few-Shot CoT prompts at test time

• Before test time: Performs inference on unlabeled training exemplars with CoT

• At test time: Retrieves similar instances to the test sample

• Improvements for Arithmetic, commonsense, and factual reasoning

Note: this one I didn’t understand the proposed architecture in practice

https://arxiv.org/pdf/2305.05181

4.7.6 Automatic Chain-of-Thought (Auto-CoT) Prompting

1. Uses Zero-Shot prompt to automatically generate chains of thought

2. Uses these generated chains to build a Few-Shot CoT prompt for a test sample

Note: GPT-3

https://arxiv.org/pdf/2210.03493

5. Decomposition Techniques

Decomposition techniques focus on breaking down complex problems into simpler sub-questions, which can significantly improve AI problem-solving abilities. While some of these techniques are similar to thought-inducing methods like Chain-of-Thought (CoT), they explicitly focus on problem breakdown.

5.1 Least-to-Most Prompting

1. Prompts the AI to break a problem into sub-problems without solving them

2. Solves sub-problems sequentially, appending responses to the prompt each time

3. Arrives at a final result

https://arxiv.org/pdf/2205.10625

5.2 Decomposed Prompting (DECOMP)

1. Uses Few-Shot prompting to teach the AI how to use certain functions (e.g., string splitting, internet searching)

2. AI breaks down the original problem into sub-problems and sends them to different functions

3. Functions are often implemented as separate AI calls

• Has shown improved performance over Least-to-Most prompting on some tasks

Note: GPT3

https://arxiv.org/pdf/2205.10625

5.3 Plan-and-Solve Prompting

• Uses an improved Zero-Shot CoT prompt: "Let's first understand the problem and devise a plan to solve it. Then, let's carry out the plan and solve the problem step by step"

• Generates more robust reasoning processes than standard Zero-Shot-CoT on multiple reasoning datasets

https://arxiv.org/pdf/2305.04091

5.4 Tree-of-Thought (ToT)

1. Creates a tree-like search problem starting with an initial problem

2. Generates multiple possible steps in the form of thoughts (as from a CoT)

3. Evaluates the progress of each step towards solving the problem (through prompting)

4. Decides which steps to continue with and keeps creating more thoughts

• Particularly effective for tasks that require search and planning

https://github.com/princeton-nlp/tree-of-thought-llm

(relevant prompts)

5.5 Recursion-of-Thought

1. Similar to regular CoT, but when it encounters a complicated problem in the reasoning chain, it sends this problem into another prompt/AI call

2. Inserts the answer from the sub-problem back into the original prompt

3. Can recursively solve complex problems, including those that might exceed the maximum context length

• Has shown improvements on arithmetic and algorithmic tasks

5.6 Faithful Chain-of-Thought

• Generates a CoT that includes both natural language and symbolic language (e.g., Python) reasoning

• Uses different types of symbolic languages in a task-dependent fashion

Note: can’t think of a practical use for this, still cool nonetheless

5.7 Skeleton-of-Thought

1. Prompts the AI to create a skeleton of the answer (sub-problems to be solved)

2. Sends these questions to an AI in parallel

3. Concatenates all the outputs to get a final response

• Focuses on accelerating answer speed through parallelization

https://arxiv.org/pdf/2307.15337

6. Ensembling Techniques

Ensembling involves using multiple prompts to solve the same problem and then aggregating these responses into a final output. These techniques often improve accuracy and reduce output variance but increase cost.

6.1 Demonstration Ensembling (DENSE)

1. Creates multiple few-shot prompts, each with a distinct subset of exemplars from the training set

2. Aggregates outputs to generate a final response

6.2 Mixture of Reasoning Experts (MoRE)

1. Creates diverse reasoning experts using specialized prompts for different reasoning types

2. Selects the best answer based on an agreement score

https://arxiv.org/pdf/2305.14628

6.3 Max Mutual Information Method

1. Creates multiple prompt templates with varied styles and exemplars

2. Selects the optimal template that maximizes mutual information between the prompt and the AI's outputs

6.4 Self-Consistency

1. Samples multiple reasoning paths for a given input

2. Selects the most consistent output as the final response

• Has shown improvements on arithmetic, commonsense, and symbolic reasoning tasks

6.5 Universal Self-Consistency

• Similar to Self-Consistency, but uses a prompt template to select the majority answer

• Helpful for free-form text generation and cases where the same answer may be output differently by different prompts

https://arxiv.org/pdf/2311.17311

6.6 Meta-Reasoning over Multiple CoTs

1. Generates multiple reasoning chains for a given problem

2. Inserts all chains into a single prompt template

3. Generates a final answer from the combined chains

https://arxiv.org/pdf/2304.13007

6.7 DiVeRSe

1. Creates multiple prompts for a given problem

2. Performs Self-Consistency for each, generating multiple reasoning paths

3. Scores reasoning paths based on each step

4. Selects a final response based on scores

6.8 Consistency-based Self-adaptive Prompting (COSP)

1. Runs Zero-Shot CoT with Self-Consistency on a set of examples

2. Selects a high agreement subset of outputs to be included in the final prompt as exemplars

3. Performs Self-Consistency again with this final prompt

6.9 Universal Self-Adaptive Prompting (USP)

• Builds upon COSP to make it generalizable to all tasks

• Uses unlabeled data to generate exemplars

• Employs a more complicated scoring function to select exemplars

• Does not use Self-Consistency

6.10 Prompt Paraphrasing

• Transforms an original prompt by changing some wording while maintaining overall meaning

• Acts as a data augmentation technique for generating prompts for an ensemble

Note: 2020, still makes sense though

7. Self-Criticism Techniques

Self-Criticism techniques involve having AI models evaluate and improve their own outputs. These methods can range from simple judgments to providing detailed feedback for improvement.

7.1 Self-Calibration

1. Prompts the AI to answer a question

2. Builds a new prompt including the question, the AI's answer, and an instruction asking whether the answer is correct

• Useful for gauging confidence levels and deciding when to accept or revise the original answer

https://arxiv.org/pdf/2207.05221

7.2 Self-Refine

1. Gets an initial answer from the AI

2. Prompts the AI to provide feedback on its own answer

3. Prompts the AI to improve the answer based on the feedback

4. Repeats the process until a stopping condition is met

• Has demonstrated improvement across reasoning, coding, and generation tasks

https://arxiv.org/pdf/2303.17651

7.3 Reversing Chain-of-Thought (RCoT)

1. Prompts the AI to reconstruct the problem based on its generated answer

2. Generates fine-grained comparisons between the original and reconstructed problems to check for inconsistencies

3. Converts inconsistencies to feedback for the AI to revise the generated answer

7.4 Self-Verification

1. Generates multiple candidate solutions with Chain-of-Thought (CoT)

2. Scores each solution by masking parts of the original question and asking the AI to predict them based on the rest of the question and the generated solution

• Has shown improvement on eight reasoning datasets

7.5 Chain-of-Verification (COVE)

1. Uses the AI to generate an answer to a given question

2. Creates a list of related questions to help verify the correctness of the answer

3. Each verification question is answered by the AI

4. All information is given to the AI to produce the final revised answer

• Improvements in various question-answering and text-generation tasks

7.6 Cumulative Reasoning

1. Generates several potential steps in answering the question

2. Has the AI evaluate these steps, deciding to accept or reject each

3. Checks whether it has arrived at the final answer

4. If not, repeats the process

• Better output in logical inference tasks and mathematical problems

8. Evaluation Techniques

LLMs can be used as evaluators, extracting and reasoning about information to assess the quality of texts or other LLM outputs. This section covers key components of evaluation frameworks using LLMs.

8.1 Prompting Techniques for Evaluation

Evaluation prompts often benefit from standard text-based prompting techniques, including:

• In-Context Learning: Provides examples of evaluations to guide the model.

• Role-based Evaluation: Assigns specific roles to the LLM to generate diverse evaluations or create a multi-agent debate setting.

• Chain-of-Thought: Improves evaluation performance by encouraging step-by-step reasoning.

• Model-Generated Guidelines: Prompts the LLM to generate its own evaluation guidelines, reducing inconsistencies from ill-defined scoring criteria.

8.2 Output Formats

The format of the LLM's evaluation output can significantly affect performance:

• Styling: Using XML or JSON formatting can improve judgment accuracy.

• Linear Scale: Simple numerical scales (e.g., 1-5, 1-10, 0-1) for scoring.

Example:

Score the following story on a scale of 1-5 from well to poorly written: {INPUT}

• Binary Score: Yes/No or True/False responses for simple judgments.

Example:

Is the following story well written at a high-school level (yes/no)?: {INPUT}

• Likert Scale: Provides a more nuanced understanding of the scale.

Example:

Score the following story according to the following scale:

Poor Acceptable Good Very Good Incredible

{INPUT}

8.3 Prompting Frameworks for Evaluation

These frameworks provide structured approaches to using LLMs for evaluation tasks:

8.3.1 LLM-EVAL

• One of the simplest evaluation frameworks

• Uses a single prompt containing:

1. A schema of variables to evaluate (e.g., grammar, relevance)

2. An instruction telling the model to output scores for each variable within a certain range

3. The content to evaluate

Notes: this works so well

https://arxiv.org/pdf/2305.13711

8.3.2 G-EVAL

• Similar to LLM-EVAL, but includes AutoCoT (Automatic Chain-of-Thought) steps in the prompt itself

• Process:

1. Generates evaluation steps according to the instructions

2. Inserts these steps into the final prompt

3. Weights answers according to token probabilities

Evaluate Coherence in the Summarization Task

You will be given one summary written for a news article.

Your task is to rate the summary on one metric.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

Coherence (1-5) - the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby ”the summary should be well-structured and well-organized. The summary should not just be a heap of related informa- tion, but should build from sentence to sentence to a coherent body of information about a topic.”

Evaluation Steps:

1. Read the news article carefully and identify the main topic and key points.

2. Read the summary and compare it to the news article. Check if the summary covers the main topic and key points of the news article, and if it presents them in a clear and logical order.

3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.

Example: Source Text: 

{{Document}} 

Summary: 

{{Summary}}

Evaluation Form (scores ONLY): 

- Coherence:

https://arxiv.org/pdf/2303.16634

8.3.3 ChatEval

• Uses a multi-agent debate framework

• Each agent has a separate role in the evaluation process

• Allows for diverse perspectives and more nuanced evaluations

These frameworks demonstrate different approaches to structuring LLM-based evaluations, from simple single-prompt methods to more complex multi-agent systems. The choice of framework can depend on the complexity of the evaluation task and the desired level of detail and diversity in the assessment.

8.4 Other Evaluation Methodologies

In addition to explicit prompting for quality assessments, there are several other methodologies for LLM-based evaluation:

8.4.1 Implicit Scoring

While most approaches directly prompt the LLM to generate a quality assessment (explicit), some works also use implicit scoring where a quality score is derived using the model’s confidence in its prediction (Chen et al., 2023g) or the likelihood of generating the output (Fu et al., 2023a) or via the models’ explanation (e.g. count the number of errors as in Fernandes et al. (2023); Kocmi and Federmann (2023a)) or via evaluation on proxy tasks (factual inconsistency via entailment as in Luo et al. (2023))

TO-DO: Implicit has worked for most of my needs, I need to check these individual papers later

8.4.2 Batch Prompting

• Evaluates multiple instances at once or the same instance under different criteria/roles. Improves compute and cost efficiency

8.4.3 Pairwise Evaluation vs Pointwise

Directly comparing the quality of two texts (Pairwise Evaluation) may lead to suboptimal results and that explicitly asking LLM to generate a score for individual summaries is the most effective and reliable method (Pointwise). The order of the inputs for pairwise comparisons can also heavily affect evaluation.

Note: relevant for summarization

https://arxiv.org/html/2406.12319v1

9. Prompting Issues: Security Concerns

https://arxiv.org/pdf/2311.16119

9.1 Types of Prompt Hacking

9.1.1 Prompt Injection

• Overrides original developer instructions in the prompt with user input

• Exploits the LLM's inability to distinguish between developer instructions and user input

9.1.2 Jailbreaking

• Gets an LLM to perform unintended actions through prompting

• Can be an architectural or training problem

9.2 Risks of Prompt Hacking

9.2.1 Data Privacy

• Training Data Reconstruction: Extracting training data from LLMs

• Prompt Leaking: Extracting the prompt template from an application

9.3 Hardening Measures

Some mitigation strategies:

9.3.1 Prompt-based Defenses

• Include instructions in the prompt to avoid prompt injection

9.3.2 Guardrails

• Rules and frameworks for guiding LLM outputs

• Range from simple classification of user input to complex dialogue managers

9.3.3 Detectors

• Tools designed to detect malicious inputs

• Often built using fine-tuned models trained on malicious prompts

• Generally more effective than prompt-based defenses

10. Prompting Issues: Alignment Concerns

10.1 Prompt Sensitivity

LLMs are highly sensitive to input prompts, with even subtle changes resulting in vastly different outputs.

10.1.1 Task Format

• Different ways of phrasing the same task can significantly alter LLM performance.

• Example: In sentiment analysis, asking to classify a review as "positive" or "negative" versus asking "Is this review positive?" (yes/no) can alter GPT-3's accuracy by up to 30% (Zhao et al., 2021b). (NOTE: I don’t think is relevant with GPT-4s anymore)

• Even logically equivalent prompts, such as altering the order of choices in multiple-choice questions, can lead to significant performance degradation. (NOTE: same, don’t think it’s relevant anymore)

10.1.2 Prompt Drift

• Occurs when the model behind an API changes over time, causing the same prompt to produce different results on the updated model.

10.2 Overconfidence and Calibration

LLMs often express overconfidence in their answers, which can lead to user overreliance on model outputs.

10.2.1 Verbalized Score

• A simple calibration technique that generates a confidence score (e.g., "How confident are you from 1 to 10").

• Efficacy is debated:

• 1 paper finds that LLMs are highly overconfident when verbalizing confidence scores, even when using self-consistency and chain-of-thought.

• In contrast, another finds that simple prompts can achieve more accurate calibration than the model's output token probabilities.

Note: I have pretty good results with confidence scores on my use cases so far

10.2.2 Sycophancy

• Refers to LLMs' tendency to agree with the user, even when contradicting their own initial output.

• LLMs are easily swayed if the user's opinion is included in the prompt (e.g., "I really like/dislike this argument").

• Questioning the LLM's original answer, strongly providing an assessment of correctness, or adding false assumptions can completely change the model output.

• Sycophancy is heightened for larger and instruction-tuned models.

• Solution: Avoid including personal opinions in prompts to prevent undue influence.

10.3 Biases, Stereotypes, and Culture

Efforts to ensure fairness and cultural sensitivity in LLM outputs include:

10.3.1 Vanilla Prompting

• Simply instructing the LLM to be unbiased in the prompt.

• Also referred to as moral self-correction.

10.3.2 Selecting Balanced Demonstrations

• Using demonstrations optimized for fairness metrics.

• Can help reduce biases in model outputs.

10.3.3 Cultural Awareness

• Injecting cultural context into prompts to help LLMs with cultural adaptation.

• Techniques include:

1. Asking the LLM to refine its own output.

2. Instructing the LLM to use culturally relevant words.

10.3.4 AttrPrompt

• Designed to avoid biases in synthetic data generation .

• Two-step process:

1. Asks the LLM to generate specific attributes important for diversity (e.g., location).

2. Prompts the LLM to generate synthetic data by varying each of these attributes.

10.4 Ambiguity

Addressing challenges posed by ambiguous questions that can be interpreted in multiple ways:

10.4.1 Ambiguous Demonstrations

• Including examples with ambiguous label sets in prompts.

• Can be automated with a retriever or done manually.

• Shown to increase In-Context Learning performance.

https://arxiv.org/pdf/2309.07900

10.4.2 Question Clarification

• Allows the LLM to identify ambiguous questions and generate clarifying questions.

• Multi-step process:

1. Generate an initial answer.

2. Classify whether to generate clarification questions or return the initial answer.

3. Decide what clarification questions to generate.

4. Generate a final answer based on clarifications.

These alignment issues highlight the complexity of ensuring LLMs behave as intended across various contexts and user needs. Addressing these concerns is crucial for the responsible and effective deployment of LLM-based systems.

https://arxiv.org/pdf/2311.09469

Ideation

I spent a lot of the first week on ideation, picking interesting things I can build alone while learning product engineering and practical LLM applications. I'm currently reading "Disciplined Entrepreneurship: 24 Steps to a Successful Startup," recommended by my friend Daniel Raffel. While not directly applicable to small solo/indie projects that I’m focused on now, it's a great resource for potential future things.

Introducing Replay

Most of my coding time this month was spent finishing Replay, a conversational journaling app I built for myself over the last few months. Journaling has been incredibly helpful for me, but I grew frustrated with existing apps. I believe LLMs can serve as a sounding board and provide context-aware probing. I'll write more about Replay in a separate post when I open it for signups. In the meantime, you can join the waitlist and see some features at replayjournal.ai.

Prompt Engineering

In Replay, I used OpenAI assistants for everything, from categorization and emotion analysis to mood detection, topic extraction, and summarization. I also have assistants for each of the 20+ journaling types and one for SEO. I spent a significant amount of time this month doing prompt engineering.

I did a lot of work on iterating on different types of summarizers for a future project.

Here are four helpful resources:

1. https://docs.anthropic.com/en/docs/prompt-engineering

2. https://platform.openai.com/docs/guides/prompt-engineering/strategy-give-models-time-to-think

3. https://www.promptingguide.ai

4. https://github.com/danielmiessler/fabric/tree/main/patterns (not a guide, but you can learn a lot from the source)

Rails Boilerplate + Tailwind

I spent a lot of time figuring out a good boilerplate for my next projects and learning Tailwind. The combination of Rails, Jumpstart Pro, and Tailwind (especially TailwindUI) has been a good choice. Tailwind has a weird learning curve and can be a pain for advanced things, but overall, I'm happy with it. Having everything in one file allows me to lean on a custom GPTs in ChatGPT and V0 for quick initial versions.

It's easy to underestimate the time spent on non-core tasks, but they always add up to weeks or look unprofessional if neglected (e.g., authentication, CDNs, servers, meta descriptions, landing pages, uploads, Stripe, etc.).

SAML + React

Last couple of weeks, I helped my friends at Koala with their SAML integration. I like their team, product and space, and I'm close with one of the founders. They're solving issues I had when helping sales at Canopy. And they have a React + Rails stack, that I wanted to see how they managed their frontend choices. They did it very pragmatically, handling the frontend quite well while benefiting from Rails on the backend (no GraphQL, model duplication or API calls). I'll write a separate post on their setup another day.

Canopy AI: Monitoring + Eval

This month, I also spent time upgrading and improving the AI leadership coach inside Canopy. It's been cool to monitor how people are using it, where it works well, and where it fails. I've started studying more about eval and assessing tools like Langsmith, Ragas, Guardrails, and OpenAI's framework. The problem is that all of these require a Python (or JS stack). So, next month, I'll likely roll out Athina.ai for Canopy to avoid introducing Python to the tech stack.

Meetups and Networking

One of the best things about being in San Francisco is the amount of people ot meet and events. I was negligent in staying connected with colleagues and friends while overworking at Canopy, but this month I started fixing that.

I also attended several meetups and virtual events, learning about AI agents, LLM-based crawling, accuracy, efficiency, RAGs, knowledge graphs, pre and post-processing, speed, and various architectures. My favorite meetups are hosted by the MLOps community, and I'm excited to attend their upcoming conference: https://www.aiqualityconference.com.

This month I also started asking for help from my girlfriend (she's an ML Engineer/Data Scientist). So she's sharing more notes from conferences, papers, etc. This paper in particular I find fascinating: https://arxiv.org/pdf/2308.10053.

Plans for Next Month: Eval + Agents

In the coming month, I plan to:

1. Open Replay for all users within the next week or two.

2. Study eval while measuring results using production data from Canopy and Replay, as well as during the prototyping phase.

3. Spend time studying both LLM theory and traditional ML.

4. Start working on a new agent-related product by the end of the month.

—

It's been an exciting and productive first month. Excited for the second!

Context: In May I left my full-time job to start a 3-month LLM learning cycle by building a few products of my own. I’m documenting my journey here. Every week I post a recap of how the week went, things I learned, etc.

Had a bit of a shorter week. I went on a camping trip to Yosemite and had no service for 3 days. Despite this, still managed to pack 3 days of 10 hours.

Here's some of the progress and learnings:

Canopy AI: JSON vs XML and streaming

I'm still helping maintain the AI Assistant I built for Canopy. It has the usual Chat experience where a user asks a question, we do a RAG search and layer a bunch of things on top.

One frustrating aspect has been GPT-4's unreliability in returning JSON. OpenAI's Completion API has a JSON mode that ensures valid JSON output, but this option isn't available on the Assistant API when using the File Retrieval tool (their built-in RAG).

Although OpenAI's built-in RAG gives me better results compared to Pinecone and PG Vector, the trade-off for Canopy's use case was losing the JSON mode.

Canopy's assistant does more than just RAG the content. It runs categorization, sensitivity checks, confidence checks, and other tasks that might trigger subsequent LLM calls or be needed in the user interface. These constraints and extra information are metadata that I used to include in a JSON (making a single API call for speed and cost).

The problem with JSON is streaming. Ideally, you want to start rendering the message as you receive the response. In the JSON, I had a markdown answer as one of the values. I'd buffer the JSON until the right value starts streaming and relay that to the user interface, stopping when the value ends. A better solution would be to have the markdown first, followed by the metadata in a structured data format (JSON, XML, etc.). After a full day of prompt engineering, I couldn't consistently get GPT-4 to return the metadata in this format after the markdown message – until GPT-4o.

Last week, I spent a day migrating from the JSON format to a structure with markdown first, followed by the metadata inside XML. Anthropic uses XML in their examples, so I've been using this tip with OpenAI as well (makes sense if you think about it, it’s easier for transformers to handle words than curly braces).

The end format looks like this:

Markdown answer…
```xml
<metadata>
…
</metadata>
```

Incorporating the ``` is another trick that took me a while to learn. When you can't use JSON mode, OpenAI will often add wrappers like these. Instead of trying to remove them in the prompt or handling failures in the app, including "shots" (examples) with the ``` as part of the answer will guide the model to always return it.

This simplified the code on our app side and made the metadata correct nearly 100% of the time. The next step is to auto-retry on the rare cases when it fails, which I haven't implemented yet.

The New Product

So far, in these 3 weeks, I've spent around 40 hours on the new product, with a significant portion of that time dedicated to boilerplate tasks that I won't need to repeat for future products, which is a positive.

While most of the AI work was already done, I couldn't resist adding some extra features at the last minute with GPT-4o. Scope creep? Why not!

I finally have a name – but I’ll share it once the landing page is up. Until now I had it under a code name, but now I have the final name, domain, and a marketing landing page 90% done.

Selecting the domain and name included initial SEO work. I think it’s BS that domains are not relevant for SEO anymore. If you are a nobody in page authority, I think it’s a good idea to make the domains very literal, so that’s how I’m starting: a short name followed by something literal.

In the 2 days I had after spending a day on Canopy, I worked on:

• Feature work:

  • Refining the AI features and adding some extra parallel LLM calls.

  • Finished an Insights page (I hate styling Heatmaps).

  • Added dark mode support while stuck in traffic driving to Yosemite.

  • Started with a Telegram bot.

• Marketing

  • Finished a landing page and registered the domain.

  • SEO: Research to find keywords I could try to rank for and figure out a strategy for evergreen content (I'll need more time here).

Before sharing the product with everyone, I still have to polish some final details, like setting up the Stripe account, deploying to the final servers, etc.

Timeline-wise, I'm thinking I need:

• 1 more full week for polishing, details, and putting up the waitlist (including boring things like Terms of Service).

• About 3 days for the initial marketing work, including backlink work, evergreen SEO-focused pages, and some community-based distribution.

Then, I'm switching attention to the second product that I want for myself (while keeping 25% of my time on this one).

Once the product is live with users, I plan to share some of the lessons more openly by writing specific posts about certain things.

For example, on this project, I've been using a mix of multiple parallel LLM calls and OpenAI batch processing. These two things could be interesting as standalone posts.

I also hope to carve out time to write separate posts about some of the product and marketing aspects whenever I can.

Week 4 Plan

This week, I'll be splitting my work into three parts:

1 day for Canopy

2 days on New Product

3 days helping a friend with his startup (I usually work a 6-day week anyway)

I'm avoiding doing any kind of external work, but in this case, I made an exception.

It's a fixed-scope project on something I'm quite familiar with, plus it's an opportunity to learn some new React on a real product designed by great programmers. On top of that, what this friend is building is an impressive product in an interesting domain with a capable team, so I'll help whenever I can. I’m estimating probably 14 days on this (let’s see how off I’ll be this time).

This will delay my LLM learning cycle plan a bit, but it's a good trade-off.

I'll continue to share my updates here regardless.

Most of the LLM criticism I see is easily fixable if people have a basic understanding of tech. OpenAI shares some blame for this, since their chat UI has become the standard but explains nothing and tries to be generic, leaning in on zero-shot to seem magical, leading to poor results out of the box.

Previously, people were using the inferior free GPT-3.5. With GPT-4o, there's no excuse for uninformed takes flooding social media and YouTube. OpenAI's terrible ChatGPT UI and closed-source GPT store are still problematic. At least Anthropic is trying to teach better prompt writing (with more effort than OpenAI).

But back to my rant...

Many users do not try to tune the system, ask poorly written questions, get unsatisfactory results, and then complain online. We're 2 years into this cycle - it's time to wise up.

By now, we should all know that these systems are few-shot predictive models that respect the chain of command. Yet users provide neither the few-shot examples nor the command it should follow. It's no wonder ChatGPT-written resumes and landing pages sound like professional BS with poor English.

To get good results, you need to:

1. Give it examples

2. Understand the chain of command

GPTs can handle Zero-shot and Few-shot (but not really)

• Zero-shot learning: performing new tasks using broad pre-training

• One-shot and few-shot learning: generalizing from 1-to-many examples to perform a task on new inputs

While GPT should handle both, its zero-shot capabilities aren't there yet beyond toying around. Always provide 2-10 examples of what you want - otherwise, you'll get predictions based on general knowledge, which means bad results.

The GPT Chain of Command

1. OpenAI and other providers set the basic rules and capabilities.

2. Developers can customize these settings for specific tasks.

3. End-user requests go through the customized settings.

For non-developers, the best way to leverage the chain of command is to create a custom GPT with a system message specifying your requirements, along with necessary examples.

If you're coding against the APIs directly, you have more options beyond just system messages, such as fine-tuning. However, in the context of ChatGPT, system messages are the primary tool available (function calls and tags can be discussed in another post).

Tip: You can use Anthropic's prompt generator to create the initial system message and examples, then paste it into ChatGPT's custom GPT (Anthropic's UI is too bad to use directly): https://x.com/danielvlopes/status/1792978707201831155

Here's a simplified version of what I use for my Blog Posts custom GPT in ChatGPT:

# Identity 
You are a draft polisher assistant. Your job is to receive long-form poorly formatted drafts and convert them to a final version maintaining the tone, and vocabulary but using fewer words, fixing the grammar, adding more spacing, better structure, and making things clearer.

# Steps to take

To do this, first carefully read through the entire blog post to understand its structure and key points. Then, go through and rewrite the polished version of each section while keeping the same titles and technical insights.

Here are some tips:

- Remove unnecessarily complex language. Replace these with plain, straightforward terms. For example:
-- Instead of "leverage", use "use"  
-- Instead of "synergize", explain what you actually mean, like "work together"
-- Instead of "disintermediation", just explain the concept in simple terms
- Break up long, run-on sentences into shorter, clearer ones
- Use contractions like "it's" and "you'll" to make it sound more natural and conversational 
- Identify the technical parts of the text and don't remove these
- Keep the tone straightforward and professional, avoiding overly casual language but ensuring it doesn't become too formal.
- Keep the personal touch and original intent of the content.
- Format the condensed version of the post with clear section breaks and spacing for readability.

Here's a list of banned words to never ever use any of the following words (or similar words): "delve", "embarked", "journey", "thrilled", "aim", "tailored", "fledgling", "shrouded", "endeavor" , "quantifiable", "empirical", "whereabouts", "debut", "keen", "craft", "tailor", "elevate", "ignite", "empower", "unleash", "horizon", "harness", "diving into", "interconnect", "enhance".

As you write the new version, make sure to maintain the same voice, tone, vocabulary, and style as the original blog post. The goal is not to rewrite it completely, but to have a slight more polished version that is easily digestible format for people short on time.

# Example 1

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 2:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

# Example 3:

Input: 
```
Original draft
```

Output: 
```
Well written version
```

Put in some effort; otherwise, you are to blame for the bad results. With early-stage technology, we can't have the luxury of expecting things to work perfectly right out of the box.

In May I left my full-time job to start a 3-month LLM learning cycle by building a few products of my own. I’m documenting my journey here. Every week I post a recap of how the week went, things I learned, etc.

Idea selection

As I said before, I'm currently focused on small indie-hacker-scoped projects. What does that mean?

I'm sorting ideas by level of scope and potential size. For the first projects, I'm avoiding super ambitious things, they need large surface area and tend to have a lot of competition. In the middle, there are “during gold rush, sell shovels” type of ideas, like AI infrastructure tools for developers where the need is clear, but so many teams are rushing to build them.

I'm looking for small things or those with seemingly bad business models that I want to use myself. When picking a market, you either solve something for yourself or others. For now, I'm choosing things I want for myself to save time on initial customer research.

However, I still want something with clusters of people easily found online so marketing isn’t an uphill battle.

To recap: I need something not hard to market with a bad TAM, small in scope, and that I want for myself (likely small apps, wrappers, or add-ons for existing players).

Perfect recipe for disaster 😅 Why?

1. It gives me time to focus on adjacent things I'll reuse later, like a clean boilerplate with UI libraries, authentication, billing, and API clients.

2. In this first month, I'm basically starting easy to rest a bit while staying creative, studying, and even having a chance to build some baby MRR (but very unlikely).

I've narrowed my list down to 5 potential ideas for now, and I’m already working on a side project that I’ve had started in the past. These ideas will be for the next project.

Week recap: progress!

Boilerplate

Rails 🚂

I'm sticking with Rails - it's stable, battle-tested, and I know it well. Not interested in Next.js yet, maybe just for marketing later.

Going with the vanilla Rails stack for now, using Hotwire instead of React for the front end. Everything has a learning curve, and modern-day React is one I don't want to tackle now. Might use React later for more interactive ideas.

Jumpstart Pro 🚀

Didn't want to waste time setting up billing (even with Stripe), accounts, authentication, 2FA, or even little stuff like pagination. Went with Jumpstart Pro - it's a good starting point with high-quality code that's easy to customize, even if I'd prefer a smaller and with less dependencies. JSP also has a lot of new things that I wasn’t familiar with like Dev Containers, so fun learning experience.

TailwindCSS + TailwindUI (Pros & Cons) 🌬️

Switched from theming Bootstrap (like I did for Canopy) to TailwindUI. Not having CSS files makes it perfect for code generators like custom ChatGPT and V0, which are great at generating Tailwind markup. TailwindUI is also very versatile and covers a lot, so I get a productive boost from the (copy, paste, and customize).

However, using TailwindUI with vanilla Rails hasn't been smooth. I find myself coding basic things because TailwindUI has no vanilla JS option for behaviors like dropdowns, carousels, and off-canvas panels. Jumpstart's Stimulus helps with some of that. Another issue is that TailwindUI's markup was meant for React components, so you often need to abstract classes into Rails helpers for repetitive use.

This week: 2 days spent to cover TailwindUI basics, 1 day to learn Jumpstart, and the learning curve of TailwindCSS every day. But now feeling quite productive already.

Brainstorming with friends

I have a lot of people reaching out to network and brainstorm ideas. Last week I spent about & hours on this.

Canopy: GPT-4o

I work 8 hours a week at Canopy, doing 1-2 hours at a time. After OpenAI launched GPT-4o, I migrated our AI Assistant:

• It's much cheaper and faster (so fast that streaming isn’t necessary anymore)

• Overall better results: I tested using our rubric of 200 questions (not a formal eval system yet). After migration, the results were mostly better.

Two small issues:

1. Acronyms: It made up some acronyms, which I fixed with prompt engineering.

2. JSON: We use OpenAI's Assistant API with their RAG tool, so we can't use "JSON MODE". I ask for JSON via prompt engineering. It tried to add a wrapper around the markdown to avoid breaking the JSON, which we fixed by adjusting the code.

The speed and price drop let us split our 200+ line prompt into two prompts, running the query sequentially:

1. Ask for the answer, present it to the user

2. Ask for additional metadata async and refresh that part of the screen (WIP, shipping next week)

The cost drop opens up the opportunity to use GPT-4 level of IQ for everything, including B2C customers with low subscription prices.

Product 1: A Lot of Work Done

As I said before, I'm already working on my first product, which I've been building since early this year a few hours a week. It's a small thing, but a useful tool for myself that benefits from LLM support.

I spent 3 days (30 hours) migrating the code to Jumpstart Pro and 4o, and adding features for launch - shooting for next weekend. I also worked on the marketing landing page.

Still to do: Complete an “insights” feature. Add an LLM-powered overview screen. TailwindUI is helping a lot with the design. So, I’m estimating ~3 to 5 more days before able to onboard users.

I’ll need to setup new servers due to significant changes from Jumpstart Pro. Setting up servers again will take likely at most one day.

Goal: For this week or next, put the product up + add a marketing landing with a waitlist, and send it to the communities I’m a member of.

The bad things

1. Poor sleep hygiene: Worked too close to bed, messing up my sleep. Getting older I need to be a lot smarter about sleep hygiene these days or the price is real.

2. Not enough exercise: Only exercised 3 times: 2x 40min half-assed weight sessions, 45min hard VO2 max virtual cycling race on Zwift. Usually do 2x that, it's important for my mental health.

3. Missed meet-ups: Trying to go to more. Had two interesting ones this week, an Agent Guidance and a Gemini vs 4o Hackathon on Saturday, and skipped both.

Next week

I'm off for 3 days on a last-minute camping trip, so this week will be short. Working this weekend to account for it. Most of my time will be split between working on Product 1 and some LLM work for Canopy.

I've been toying with LLMs since end of 2022, trying to create an AI Assistant for my company, Canopy. The tech wasn’t quite there until GPT-4 came along. Last week, after launching our AI Assistant in alpha, I decided to leave my job to focus on developing my own ideas. I’ll be documenting my journey here as I go, and this is the recap of my first week.

Why now?

With the last couple of years' progress, so many of the traditional ML projects have been (or could be) replaced by language models, making things easier compared to just three years ago. Things like unstructured data, classification, sentiment analysis, NLP, speech, and much more are now all manageable by one person or a tiny team.

I think there are four ways to benefit from what is happening:

1. Big Tech working on foundational infrastructure or large problems.

2. Moonshot projects challenging existing players like Perplexity vs Google.

3. Vertical applications (or what people are now calling "Applied AI").

4. Indie hackers focused on things that don't look venture-scale at first glance.

For now, for these next three months, I'm personally interested in item 4. Maybe later I'll try to join a team working on the other ones.

Setting a 3-month deadline

Having a short deadline helps keep me motivated and accountable. It pushes me to focus on practical, achievable goals.

Since I work in product and not ML or research, I'm interested in end-use application. There's a ton of effort involved in making LLMs (unstable, unreliable & expensive tech) work smoothly in a finished product.

For instance, if I were only studying, factors like accuracy, cost, or user experience would be glossed over. But if I need to ship and charge for something, that’s the opposite.

Three months, with my skill set, I think is a doable but still challenging timeline to ship at least two products (my stretch goal is six).

Now, to the first-week recap

I had 5 goals for this past week:

1. Ship my personal site, start a Substack, clean up social media.

2. Get a boilerplate codebase for my projects (sign-up, subscription, etc).

3. Organize and research my ideas: pick 3 to 6.

4. Canopy: monitor the Alpha launch of our AI Assistant.

5. Bonus: Ship a WIP project that I had already.

✅ Goal 1: Personal site, Substack & social media

Personal site: Over the weekend, I started working on a personal site that I’d need to send to some people. I also used it to play with Next, Vite, Astro, Tailwind, and Shadcn. All things not part of Canopy stack.

I went with Next + TailwindUI. Site is live here: https://daniellopes.dev

For my products will be using just Tailwind and my usual Rails.

Substack: Working with the garage door open is something I always wanted to do for Canopy, but I left that for my co-founder. So, last week I started this, and I posted the first post: Using LLMs to parse 7 years of stand-up updates (which was an exercise in trying different summarization approaches as well).

Writing a ton is also part of one of my ideas that I might work on next month.

Social media: To stay more informed about things happening in San Francisco (meetups, hackathons, etc). So spent a few hours making list of people I want to follow more closely. Social media isn't all just noise; it can be valuable.

🟨 Goal 2: Organize all my ideas

I made a lot of progress on this goal, but not enough to confidently have three solid ones. I'll keep working on this next week. Here's the gist:

I keep a list of ideas, logging anything that comes to mind (features or products). I always have anywhere around 100 to 200 items.

This week, primary goal was to organize my list, considering these 4 criteria:

• Is it a newly viable thing? And is it flooded with people doing good work already? (Areas like Knowledge Base for Companies or Customer Support are already saturated.)
  

• Do I want this for myself? Building something for others demands a lot of research, which isn't feasible within a three-month timeline.
  

• Does it have commercial potential?

  • If no commercial potential, the aim should be for learning or personal use; don't waste time on unviable or tarpit ideas. Do just enough to meet the goal and move on. At the end of the day, I want people to use the things I build.

  • If there's commercial potential, do I have a unique perspective on solving the challenges? Then do the usual analysis—pricing, customer base needed for a MRR, competitors, etc.
    

• Does it have the potential to find a single feature that could be a lot better than existing methods? And can it be developed as an add-on for existing players or does it need to be a replacement? (replacements are less interesting with this timeline).
  

• What’s the marketing effort? Can I market it alone? If it's B2B, can I identify a sustainable group of people to target? If it's B2C, can I come up with marketing strategies that don't rely on ads or require excessive time?

Many ideas I had on the list from last year had no credible competitors six months ago, but now have promising ones. So, I ditched those. While I don't mind competition when working with more resources, working solo, I don’t need my back against the wall all the time.

✅ Goal 3: Boilerplate

Initially, I planned to use much of Canopy's codebase for non-proprietary elements and create a standard boilerplate things like sign-up, subscriptions, and admin (which would let me contribute back improvements).

However, given my time constraints, I'm reluctant to spend excessive time cleaning things like our complex billing system that covers B2B, B2C, and enterprise solutions.

I've decided to go with Jumpstart Pro. There are several things I plan to remove/change, but for now, it's enough. I’ve also wanted to learn Tailwind for a while, so it was nice that Jumpstart Pro came with Tailwind already.

I’m also sticking to Rails (The One Person Framework ™️).

I’ve been tempted to make the jump to Python because of how readily available tooling is for their ecosystem, but the fastest way to ship nothing is to pick a bunch of things you are not expert on. It’s too easy to underestimate all the ecosystem details that you need to know in order to ship a product that is stable enough to require nearly no-maintenance.

In this case, my goal is to focus on the behavior of LLMs (different models, prompt architecture, cost, reliability, etc.) and not mess around with different frontend or backend technologies.

✅ Goal 4: Canopy - Monitor the Alpha of AI Assistant

We launched our AI Assistant in Alpha last week. So, I focused on getting visibility for how the system is performing. We haven't fully committed to an evaluation strategy yet. For now, we managed to see how satisfied people are, along with some basic analytics and reporting on content that we didn't have for RAG.

For “eval” this looks very promising: https://athina.ai

I’ve seen their product at a meet-up in SF last week and found it a better fit for our needs than other tools I'd considered (open source solutions like OpenAI’s or Ragas.)

❌ Goal 5: Ship a WIP product

I didn't succeed with this one.

Last year, I started working on a product that solves a personal use. None of the business calculations add up (cost per person vs what I can charge, TAM, competitors, etc.), but it meets my criteria of being something I want to use and something I can learn from.

I've already learned a lot from this project. It's what I worked on over the weekends last year to learn different models, prompting or RAG architecture without getting burnt out from just working at Canopy. It's ~80% done, but now I just need the last stretch to make it possible for others to sign up & buy it so I'm not paying for other people's API calls.

I'll probably need another three days to get it ready for other users. Since I chose Jumpstart Pro, I'm going to migrate the product to it and see how that goes. So, this will carry over into Week 2.

Plan for Week 2

Switching jobs, picking ideas, setting up the basics, and wrapping up existing projects is kind of like tying up loose ends. It's not the most thrilling, but it's still important.

For next week, I have 3 main goals: Launch (1) and market (2) the project I just mentioned, and to do more research on my ideas (3).

Custom GPTs is the feature from ChatGPT that makes me use it more than anything else (more than Claude, Perplexity, or Gemini). I have a handful Custom GPTs that I use every day, and I create a bunch for specific needs as well.

One of my favorite use cases for one of my recurring Custom GPTs is to brainstorm different solutions before I start coding (this one is using a Rails-specific GPT one). I pretty much will start planning my work there. It will often give me good suggestions or at least provide me with some good naming for things. Then at the end a finish by asking for a review as well.

In this example, I know what different patterns I might want to use here. I could do it myself no problem, but I find it great to be able to see a draft version of all the different ideas in 5 seconds first and then decide whether to try to do it in my head, picking just one and spending 20 minutes executing on it.

The request

I need to reuse these lines on a different controller. What would you suggest? Maybe should I extract a service or PORO class? or a concern for controllers?

@learning_path = current_person.personal_learning_path
@learning_path_modules = @learning_path.training_modules.includes(:lessons) if @learning_path
@learning_path_progress = @learning_path.progress_for(current_person) if @learning_path
@training_modules = load_training_modules

---

def load_training_modules
  if current_organization_training_subscription&.trial?
    TrainingModule.standard_order_for_trial
  else
    TrainingModule.preferred_order_for(
      current_person,
      order_by_preference: "recommendations",
      exclude_seen_modules: false,
      consider_watched_lessons: true,
      alternate_results: false
    )
  end
end

====HERES THE FULL CONTROLLER=====

class TrainingModulesController < TrainingBaseController
  menu_item :training_modules
  before_action :initial_breadcrumbs

  def index
    @assessment = current_person.last_assessment_with_self_response_complete
    current_person.generate_personal_learning_path_if_needed!

    @learning_path = current_person.personal_learning_path
    @learning_path_modules = @learning_path.training_modules.includes(:lessons) if @learning_path
    @learning_path_progress = @learning_path.progress_for(current_person) if @learning_path

    @training_modules = load_training_modules
    @display_tour = !current_navigation_preference.learning_paths_tour_closed?
  end

  def show
    @training_module = TrainingModule.includes(:lessons).find(params[:id])
    @progress = TrainingModuleProgress.new(current_person, @training_module)
    @lesson = @progress.next_available_lesson || @training_module.lessons.first

    current_person.training_history_entries.track(request.url, @training_module)

    redirect_to training_module_lesson_path(@training_module, @lesson)
  end

  private

  def initial_breadcrumbs
    breadcrumb.add "Modules", training_modules_path
  end

  def load_training_modules
    if current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        current_person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

GPT Response

Given the repetitive use of the code involved in fetching and preparing `@learning_path` and related entities across different controller actions, extracting these operations into a reusable component is a good idea. This will help maintain DRY (Don't Repeat Yourself) principles and ensure cleaner and more maintainable code. Here are a few options to consider:

### 1. Service Object

A service object is ideal for encapsulating business logic that doesn't naturally fit within a model or a controller. Since your code involves preparing data for views based on user-specific and contextual business rules, a service object could be a good fit.

# app/services/learning_path_service.rb
class LearningPathService
  def initialize(person)
    @person = person
  end

  def fetch_data
    learning_path = @person.personal_learning_path
    {
      learning_path: learning_path,
      learning_path_modules: learning_path&.training_modules&.includes(:lessons),
      learning_path_progress: learning_path&.progress_for(@person),
      training_modules: load_training_modules
    }
  end

  private

  def load_training_modules
    if @person.current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        @person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

Usage in a controller:

def index
  service = LearningPathService.new(current_person)
  data = service.fetch_data
  @learning_path, @learning_path_modules, @learning_path_progress, @training_modules = data.values_at(:learning_path, :learning_path_modules, :learning_path_progress, :training_modules)
  @display_tour = !current_navigation_preference.learning_paths_tour_closed?
end

### 2. Concern

If the logic is tightly coupled with ActiveRecord objects or used across multiple controllers, creating a concern might be a better approach. This keeps controller-specific logic within the context of controllers.

# app/controllers/concerns/learning_path_loadable.rb

module LearningPathLoadable
  extend ActiveSupport::Concern

  def load_learning_path_data
    @learning_path = current_person.personal_learning_path
    @learning_path_modules = @learning_path&.training_modules&.includes(:lessons)
    @learning_path_progress = @learning_path&.progress_for(current_person)
    @training_modules = load_training_modules
  end

  private

  def load_training_modules
    if current_organization_training_subscription&.trial?
      TrainingModule.standard_order_for_trial
    else
      TrainingModule.preferred_order_for(
        current_person,
        order_by_preference: "recommendations",
        exclude_seen_modules: false,
        consider_watched_lessons: true,
        alternate_results: false
      )
    end
  end
end

Usage in a controller:

class TrainingModulesController < TrainingBaseController
  include LearningPathLoadable

  def index
    load_learning_path_data
    @display_tour = !current_navigation_preference.learning_paths_tour_closed?
  end
end

### Decision

The choice between a service object and a concern depends on how closely the logic is tied to the controller's responsibilities. If the logic is more about data manipulation and less about HTTP-specific operations, a service object is more suitable. If it's about setting up controller instance variables and integrates closely with controller lifecycle (like before actions), a concern might work better. In your case, either approach would streamline your code and enhance maintainability, with a slight preference for a service object given the nature of the logic involved.

Final decision

I ended up with the concern approach with a code that is very similar to the original GPT suggestion.

Custom GPT “source code”

This custom GPT is the one I use for Canopy setup for the following use cases (and a has few examples of our codebase inside):

I've been looking back into my daily stand-up updates and turning things into monthly or quarterly reflections under the #journal tag here. This note is from my first quarter at Canopy after doing some parsing and summarization over the last 7 years.

The company was bootstrapped and didn't have a technical co-founder yet. My partner had done a great job so far with the resources she had, but now there was some foundational work to do if we wanted to see some growth.

Month 1

In my first week, I set up Segment to integrate our marketing and sales tools like CRM and Drip Campaigns, and to lay the groundwork for a more robust analytics system.

Our in-app analytics was slow and crashing. So it had been turned off and we had no reports for things like engagement or revenue per account. Sales had somewhat long cycles driven by contracts with a high upfront cost; we had a pay once per email address pricing at the time that was likely masking the real MRR/ARR.

During the first month, my goal was to improve admin performance and get our reports running again. I did this without setting up a new BI tool, although connecting Segment to a warehouse using BI tools like Hex or Mode would have been smarter.

I also streamlined our sales process by integrating Freshsales, added an Intercom chat for user feedback, and upgraded to Rails 5.

My main focuses in the first 30 days were:

1. Lowering the error rate.

2. Updating our tech stack.

3. Implementing key marketing, sales, and support tools.

4. Getting visibility on key metrics like retention, engagement, and LTV.

Month 2

We launched a Knowledge Center feature within the product—a collection of eBooks that users can read directly in the app. This required building a simple CMS with markdown support and PDF download functionality. Interestingly, this was the initial feature that would validate the interest in content as a service, which we fully embraced in 2023.

I also began building our Slack integration. At the time, our system relied heavily on email, and most customers had migrated to Slack. Investing in Slack integration proved to be a good decision for engagement.

Additionally, I helped with sales enablement initiatives, like deploying a read-only demo environment. I also supported some marketing efforts like a hosting platform for an email-based mini-course.

Month 3 and 4

We shipped our Slack integration and improved our billing system to enable future recurring pricing tests. We also fixed several billing issues.

It became clear that engagement was strong in the first months but tapered off (it made sense for the feature we had). I surveyed customers and analyzed user session recordings to understand this better.

We realized our product wasn't ready for a dedicated sales role yet. It was time for founding sales. To achieve better product-market fit, we unfortunately had to let go of our sales person, and my co-founder took over sales responsibilities to enable founder-led sales.

--

Reflection

In my opinion, product-market fit (PMF) works in levels. What we had was more than sufficient for a great small business of 2 making up to $200k each, but it wasn't enough for our SaaS compound growth ambitions. We didn't have PMF for the next level. And trying to find PMF, you don’t need things like sales quotas or bonuses interfering with the process.

Also, get visibility as soon as you start getting some volume. When you are on low hundreds of users, you may not need full-fledged BI yet. Tools like Smartlook or a basic activity feed per account might be enough. However, nowadays, I would add BI much sooner using Hex because it's so easy to set up and maintain.

I think everyone should have a Year Review doc and take note quarterly of all your work (projects, impact, launches, metrics, etc.) so you can use this in interviews and for your resume.

This is a problem that I have myself. I've spent 7 years at Canopy, wrote most of the 200kloc of our codebase, helped a lot with our UI design, was involved in all product and hiring decisions, helped fundraise, etc.

But my LinkedIn is empty. I honestly don't even remember what I did.

The thing is that for 7 years I've used Canopy Perform to record daily standups. I have a journal of everything. Now, how do I make sense of it?

Summarization

So I decided to summarize everything with GPT4 and Claude Opus and see the differences. The steps:

1. Exported the CSV with all my Heartbeats from Canopy Perform. The data was a row for every day containing HTML with lists, strong tags, and divs of what I did on that day, what I did the day before, and sometimes reflections and challenges.

2. Used ChatGPT with Code Interpreter to rewrite the CSV, combining every day into a new CSV that had weeks instead of days, and the value of each week was all the days of that week combined.

3. Wrote this script to call OpenAI gpt-4-turbo-preview and summarize every week while getting rid of the HTML (tried with GPT 3.5, and as usual, it's useless). The output wasn't perfect – I should have added more guidance on vocabulary to cut the BS words, but good enough.

4. Then I wanted to manually go over every month and try to reflect/remember what I did. For that, I decided to test the differences between a Custom GPT in ChatGPT, OpenAI Playground, and Claude Opus with the same system message.

Comparing Claude, Custom GPT and OpenAI API

Here's the comparison of each on my 1st full month at Canopy:

This was the System Message (same used for all), and here’s the input data.

Note: the summary was generated by GPT4 with low guidance, so there's some BS vocabulary that I would never use.

Claude console result

All default settings: temperature 0, claude-3-opus-2024, max tokens sample 1k

Title: April 2017: Focusing on Admin Panel, Rails 5 Update, and Slack Integration:

In April, I focused on improving our Admin panel, particularly the statistics section, which was deactivated due to performance issues. I also worked on updating our system to Rails 5 to keep pace with upcoming updates and facilitate a smoother transition to Rails 5.1 in the future. 

Additionally, I began integrating Slack into our workflow to enhance our communication and collaboration processes. Although I had to postpone some administrative tasks, such as completing revenue and question popularity reports, I prioritized these system updates and integrations to ensure our technology remains current and efficient.

ChatGPT with a Custom GPT:

**April 2017: System Enhancements and Administrative Focus**

This month involved significant focus on improving our Admin panel, particularly the statistics section, due to performance issues. I managed the transition from CRM and Intercom management to addressing these concerns, aiming to restore and optimize this vital tool for our team’s decision-making.

Following this, I tackled pending administrative tasks and began updating our system to Rails 5, balancing these priorities effectively. The update was crucial for keeping pace with new technology and was strategically planned to ensure a smooth transition to future versions.

I also initiated integration with Slack, enhancing our workflow efficiency, although this task is ongoing. Overall, the month was about enhancing system performance and efficiency while managing routine administrative duties.

OpenAI Playground (GPT-4-turbo-preview)

Default temperature 0.5 with gpt-4-turbo but max tokens increased to 1k

### Monthly Work Summary: April 2017

This month, I focused on various critical areas in our system administration. I began by addressing performance issues in the statistics section of our Admin panel, aiming to enhance its efficiency and restore its functionality.

Following that, I handled pending administrative tasks and initiated the upgrade of our system to Rails 5. This update was important to keep our system compatible with upcoming versions and improve overall performance.

Additionally, I planned to integrate Slack into our workflow to streamline communications. Each of these tasks was vital for maintaining and improving our operational capabilities.

I think my preference order here is:

1. Claude Opus

2. OpenAI Playground with GPT-4-turbo

3. ChatGPT with Custom GPT

What I actually did

Title: April 2017: Focusing on Admin Panel, Rails 5 Update, and Slack Integration

In April, I focused on improving our Admin panel, particularly the statistics section, which was turned off due to performance issues. 

I also updated us to Rails 5 to keep facilitate a smoother transition to Rails 5.1 in the future. Additionally, I began the work on the Slack integration feature. Slack took priority over some of the extra Admin things I wanted to do like adding a Revenue and Engagement reports.

Some of these things got lost in the first pass of summarization that GPT-4 did on the Python script above.

Conclusion

After running through all my ~88 months at KYT/Canopy, I think this was a very helpful reflection exercise to see all the things I did. It helped me reflect on my mistakes and on things I did right.

Moving forward, I plan to keep a monthly journal like this and keep it up-to-date here on Substack under the Journal tag. I’ll share a post a day in retrospect, covering all these last years. Here’s the first one:

I've celebrated multiple achievements at work. We rebranded KYT to Canopy in just three weeks, launched our iOS app, and shipped our AI Assistant. But, last week was also my last full-time at Canopy.

Things weren't always easy. Going after the ambitious goal of helping people become better managers at scale with a bootstrapped small team required a lot of effort, and I am so proud of what we accomplished.

When I joined, for a long time, it was just Claire and me. We grew it into a business making 7-figures in annual revenue, an amazing team of 8 people, and reached tens of thousands of users. Working with Claire and the team has been amazing.

But after seven years, it’s time for something new. So I’m moving into a technical advisor role at Canopy.

I’m not exactly sure what comes next. One thing I am sure about is that I want to work with LLMs a bit more.

My last project at Canopy was our AI assistant. It was a blast to work on, and I learned a lot. This project is something I've been working on and off since GPT-3, and now it’s up and running beautifully.

There's so much potential. For now, I’ll focus on learning more about similar applications. I learn best by doing and I have several ideas involving LLMs—some more personal, others with maybe business potential.

I’ve set aside three months to work on these projects. If I like the direction, I might try to extend for another 3 and see from there.

I'm going to share my experiences on Substack / Twitter / LinkedIn.

This is the first time since moving to the US that I have changed roles without another job lined up. It feels weird but also exciting.