Welcome

By definition, Prompty is a new asset class and format for LLM prompts that aims to provide observability, understandability, and portability for developers. In this workshop, we explore what this means from core concepts to prompty prototyping in a series of hands-on labs.

Learning Objectives

The workshop is structured as a series of hands-on labs taking you from core concepts to prompt evaluation and composition, with code. On completion, you should be able to describe:

1. What Prompty is and the benefits it offers
2. How to create and execute a simple Prompty asset
3. How to choose and configure models for your prompt
4. How to engineer prompts for different scenarios
5. How to instrument Prompty assets for observability
6. How to evaluate prompts to assess quality and safety
7. How to compose Prompty assets for complex flows

Pre-Requisites

To complete these labs you will need:

1. A personal GitHub account (create one for free)
2. A Microsoft Azure account (create one for free)
3. Basic familiarity with Python syntax and tools
4. Basic familiarity with Generative AI concepts
5. Basic familiarity with Visual Studio Code tooling

Learning Modules

The workshop is organized into modules, each focused on one topic and containing a series of hands-on lab units. Click the module title to see details of the units.

1. Core Concepts

• Prompty Specification
• Prompty Tooling
• Prompty Runtime

2. Prompty Quickstart

• Create a new Prompty asset
• Configure asset with a model
• Configure asset with a prompty template
• Execute asset with a hardcoded input

3. Prompt Engineering

• By modifying model parameters
• By modifying system context
• By modifying prompt instructions
• By adding grounding context

4. Model Choice

Refactor prompty asset to use models sourced from:

• GitHub Marketplace
• Open AI
• Azure OpenAI
• Azure AI Model Catalog (serverless API)
• LlamaIndex

5. Prompty Observability

• Instrumenting assets for tracing
• Viewing trace logs in the Prompty UI

6. Prompty Testing

• With a sample data file
• With hardcoded data inputs
• With interactive data inputs from CLI

7. Prompty Composition

• Chain prompty assets for evaluation
• Chain prompty assets for orchestration

8. Prompty Evaluation

• By writing custom evaluators
• By running evaluators with test data
• By assessing quality with test data
• By running evaluators with adversarial data
• By assessing safety with adversarial data

Learning Resources

Bookmark and explore these resources:

• Documentation - official documentation, in-progress.
• Repository - open-source repository.
• Prompty in Promptflow - with examples

Watch the Microsoft Build 2024 session for a practical tour of end-to-end usage.