# Component API Reference

TIP

Our TypeScript component API is in beta. If you're interested in developing TypeScript components and providing feedback, see our TypeScript docs.

This document was created to help developers author and use Pipedream components. Not only can you develop sources (workflow triggers) and actions using the component API, but you can also develop Node.js steps right in your workflows - without leaving your browser! You can publish components to your account for private use, or contribute them to the Pipedream registry for anyone to run.

While sources and actions share the same core component API, they differ in both how they're used and written, so certain parts of the component API apply only to one or the other. This section of the docs explains the core differences. When this document uses the term "component", the corresponding feature applies to both sources and actions. If a specific feature applies to only sources or actions, the correct term will be used.

If you have any questions about component development, please reach out in our community (opens new window).

Overview
Component API
Managing Components
- Managing Sources
- Managing Actions
Source Lifecycle
Source Event Lifecycle

# Overview

# What is a component?

Components are Node.js modules that run on Pipedream's serverless infrastructure.

Trigger Node.js code on HTTP requests, timers, cron schedules, or manually
Emit data on each event to inspect it. Trigger Pipedream hosted workflows or access it outside of Pipedream via API
Accept user input on deploy via CLI, API, or UI (opens new window)
Connect to 1,400+ apps (opens new window) using Pipedream managed auth
Use most npm packages with no npm install or package.json required
Store and retrieve state using the built-in key-value store

# Quickstarts

To help you get started, we created a step-by-step walkthrough for developing both sources and actions. We recommend starting with those docs and using the API reference below as you develop.

# Differences between sources and actions

Sources and actions share the same component API. However, certain features of the API only apply to one or the other:

Actions are defined with type: action (see the docs on the type property). Sources require no type property be set. Components without a type are considered sources.
Sources emit events using this.$emit, which trigger linked workflows. Any features associated with emitting events (e.g., dedupe strategies) can only be used with sources. Actions return data using return or $.export, which is made available to future steps of the associated workflow.
Sources have access to lifecycle hooks, which are often required to configure the source to listen for new events. Actions do not have access to these lifecycle hooks.
Actions have access to a special $ variable, passed as a parameter to the run method. This variable exposes functions that allow you to send data to destinations, export data from the action, return HTTP responses, and more.
Sources can be developed iteratively using pd dev. Actions currently cannot (please follow this issue (opens new window) to be notified of updates).
You use pd deploy to deploy sources to your account. You use pd publish to publish actions, making them available for use in workflows.
You can attach interfaces (like HTTP endpoints, or timers) to sources. This defines how the source is invoked. Actions do not have interfaces, since they're run step-by-step as a part of the associated workflow.

# Getting Started with the CLI

Several examples below use the Pipedream CLI. To install it, follow the instructions for your OS / architecture.

See the CLI reference for detailed usage and examples beyond those covered below.

# Example Components

You can find hundreds of example components in the components/ directory of the PipedreamHQ/pipedream repo (opens new window).

# Component API

# Component Structure

Pipedream components export an object with the following properties:

export default {
  name: "",
  key: "",
  type: "",
  version: "",
  description: "",
  props: {},
  methods: {},
  hooks: {
    async activate() {},
    async deactivate() {},
    async deploy() {},
  },
  dedupe: "",
  async run(event) {
    this.$emit(event);
  },
};

Property	Type	Required?	Description
`name`	`string`	required	The name of the component, a string which identifies components deployed to users' accounts. This name will show up in the Pipedream UI, in CLI output (for example, from `pd list` commands), etc. It will also be converted to a unique slug on deploy to reference a specific component instance (it will be auto-incremented if not unique within a user account).
`key`	`string`	recommended	The `key` uniquely identifies a component within a namespace. The default namespace for components is your account. When publishing components to the Pipedream registry, the `key` must be unique across registry components and should follow the pattern: `app_name_slug`-`slugified-component-name`
`type`	`string`	required	When publishing an action, `type: "action"` is required. When publishing a source, use `type: "source"`.
`version`	`string`	required	The component version. There are no constraints on the version, but semantic versioning (opens new window) is required for any components published to the Pipedream registry.
`description`	`string`	recommended	The description will appear in the Pipedream UI to aid in discovery and to contextualize instantiated components
`props`	`object`	optional	Props are custom attributes you can register on a component. When a value is passed to a prop attribute, it becomes a property on that component instance. You can reference these properties in component code using `this` (e.g., `this.propName`).
`methods`	`object`	optional	Define component methods for the component instance. They can be referenced via `this` (e.g., `this.methodName()`).
`hooks`	`object`	optional (sources only)	Hooks are functions that are executed when specific component lifecycle events occur.
`dedupe`	`string`	optional (sources only)	You may specify a dedupe strategy to be applied to emitted events
`run`	`method`	required	Each time a component is invoked (for example, via HTTP request), its `run` method is called. The event that triggered the component is passed to `run`, so that you can access it within the method. Events are emitted using `this.$emit()`.

# Props

Props are custom attributes you can register on a component. When a value is passed to a prop attribute, it becomes a property on that component instance. You can reference these properties in component code using this (e.g., this.propName).

Prop Type	Description
User Input	Enable components to accept input on deploy
Interface	Attaches a Pipedream interface to your component (e.g., an HTTP interface or timer)
Service	Attaches a Pipedream service to your component (e.g., a key-value database to maintain state)
App	Enables managed auth for a component

# User Input Props

User input props allow components to accept input on deploy. When deploying a component, users will be prompted to enter values for these props, setting the behavior of the component accordingly.

# General

Definition

props: {
  myPropName: {
    type: "",
    label: "",
    description: "",
    options: [], // OR async options() {} to return dynamic options
    optional: true || false,
    propDefinition: [],
    default: "",
    secret: true || false,
    min: <integer>,
    max: <integer>
  },
},

Property	Type	Required?	Description
`type`	`string`	required	Value must be set to a valid `PropType` (see below). Suffix with `[]` (e.g. `string[]`) to denote array of that type (if supported).
`label`	`string`	optional	A friendly label to show to user for this prop. If a label is not provided, the `propName` is displayed to the user.
`description`	`string`	optional	Displayed near the prop input. Typically used to contextualize the prop or provide instructions to help users input the correct value. Markdown is supported.
`options`	`string[]` or `object[]` or `method`	optional	Provide an array to display options to a user in a drop down menu. `[]` Basic usage Array of strings. E.g., `['option 1', 'option 2']` `object[]` Define Label and Value `[{ label: 'Label 1', value: 'label1'}, { label: 'Label 2', value: 'label2'}]` `method` Dynamic Options You can generate options dynamically (e.g., based on real-time API requests with pagination). See configuration details below.
`useQuery`	`boolean`	optional	Use in conjunction with Dynamic Options. If set to `true`, the prop accepts a real-time query that can be used by the `options` method to obtain results according to that query.
`optional`	`boolean`	optional	Set to `true` to make this prop optional. Defaults to `false`.
`propDefinition`	`[]`	optional	Re-use a prop defined in an app file. When you include a prop definition, the prop will inherit values for all the properties listed here. However, you can override those values by redefining them for a given prop instance. See propDefinitions below for usage.
`default`	`string`	optional	Define a default value if the field is not completed. Can only be defined for optional fields (required fields require explicit user input).
`secret`	`boolean`	optional	If set to `true`, this field will hide your input in the browser like a password field, and its value will be encrypted in Pipedream's database. The value will be decrypted when the component is run in the execution environment. Defaults to `false`. Only allowed for `string` props.
`min`	`integer`	optional	Minimum allowed integer value. Only allowed for `integer` props..
`max`	`integer`	optional	Maximum allowed integer value . Only allowed for `integer` props.

PropTypes

`PropType`	Array Supported	Supported in Sources?	Supported in Actions?	Custom properties
`app`		✓	✓	See App Props below
`boolean`	✓	✓	✓
`integer`	✓	✓	✓	- `min` (`integer`): Minimum allowed integer value. - `max` (`integer`): Maximum allowed integer value.
`string`	✓	✓	✓	- `secret` (`boolean`): Whether to treat the value as a secret.
`object`		✓	✓
`any`			✓
`$.interface.http`		✓
`$.interface.timer`		✓
`$.service.db`		✓

Usage

Code	Description	Read Scope	Write Scope
`this.myPropName`	Returns the configured value of the prop	`run()` `hooks` `methods`	n/a (input props may only be modified on component deploy or update via UI, CLI or API)

Example

Following is an example source that demonstrates how to capture user input via a prop and emit it on each event:

export default {
  name: "User Input Prop Example",
  version: "0.1",
  props: {
    msg: {
      type: "string",
      label: "Message",
      description: "Enter a message to `console.log()`",
    },
  },
  async run() {
    this.$emit(this.msg);
  },
};

To see more examples, explore the curated components in Pipedream's GitHub repo.

# Advanced Configuration

# Async Options (example (opens new window))

Async options allow users to select prop values that can be programmatically-generated (e.g., based on a real-time API response).

async options({
  page,
  prevContext,
  query,
}) {},

Property	Type	Required?	Description
`options()`	`method`	optional	Typically returns an array of values matching the prop type (e.g., `string`) or an array of object that define the `label` and `value` for each option. The `page` and `prevContext` input parameter names are reserved for pagination (see below). When using `prevContext` for pagination, it must return an object with an `options` array and a `context` object with a `nextPageToken` key. E.g., `{ options, context: { nextPageToken }, }`
`page`	`integer`	optional	Returns a `0` indexed page number. Use with APIs that accept a numeric page number for pagination.
`prevContext`	`string`	optional	Returns a string representing the context for the previous `options` execution. Use with APIs that accept a token representing the last record for pagination.
`query`	`string`	optional	Returns a string with the user input if the prop has the `useQuery` property set to `true`. Use with APIs that return items based on a query or search parameter.

Following is an example source demonstrating the usage of async options:

export default {
  name: "Async Options Example",
  version: "0.1",
  props: {
    msg: {
      type: "string",
      label: "Message",
      description: "Select a message to `console.log()`",
      async options() {
        // write any node code that returns a string[] or object[] (with label/value keys)
        return ["This is option 1", "This is option 2"];
      },
    },
  },
  async run() {
    this.$emit(this.msg);
  },
};

# Prop Definitions (example (opens new window))

Prop definitions enable you to reuse props that are defined in another object. A common use case is to enable re-use of props that are defined for a specific app.

props: {
  myPropName: {
    propDefinition: [
      app,
      "propDefinitionName",
      inputValues
    ]
  },
},

Property	Type	Required?	Description
`propDefinition`	`array`	optional	An array of options that define a reference to a `propDefinitions` within the `propDefinitions` for an `app`
`app`	`object`	required	An app object
`propDefinitionName`	`string`	required	The name of a specific `propDefinition` defined in the corresponding `app` object
`inputValues`	`object`	optional	Values to pass into the prop definition. To reference values from previous props, use an arrow function. E.g.,: `c => ({ variableName: c.previousPropName })` See these docs for more information.

Following is an example source that demonstrates how to use propDefinitions.

const rss = {
  type: "app",
  app: "rss",
  propDefinitions: {
    urlDef: {
      type: "string",
      label: "RSS URL",
      description: "Enter a URL for an RSS feed.",
    },
  },
};

export default {
  name: "Prop Definition Example",
  description: `This component captures an RSS URL and logs it`,
  version: "0.1",
  props: {
    rss,
    url: { propDefinition: [rss, "urlDef"] },
  },
  async run() {
    console.log(this.url);
  },
};

# Referencing values from previous props

When you define a prop in an app file, and that prop depends on the value of another prop, you'll need to pass the value of the previous props in a special way. Let's review an example from Trello (opens new window), a task manager.

You create Trello boards for new projects. Boards contain lists. For example, this Active board contains two lists:

In Pipedream, users can choose from lists on a specific board:

Both Board and Lists are defined in the Trello app file:

board: {
  type: "string",
  label: "Board",
  async options(opts) {
    const boards = await this.getBoards(this.$auth.oauth_uid);
    const activeBoards = boards.filter((board) => board.closed === false);
    return activeBoards.map((board) => {
      return { label: board.name, value: board.id };
    });
  },
},
lists: {
  type: "string[]",
  label: "Lists",
  optional: true,
  async options(opts) {
    const lists = await this.getLists(opts.board);
    return lists.map((list) => {
      return { label: list.name, value: list.id };
    });
  },
}

In the lists prop, notice how opts.board references the board. You can pass opts to the prop's options method when you reference propDefinitions in specific components:

board: { propDefinition: [trello, "board"] },
lists: {
  propDefinition: [
    trello,
    "lists",
    (configuredProps) => ({ board: configuredProps.board }),
  ],
},

configuredProps contains the props the user previously configured (the board). This allows the lists prop to use it in the options method.

# Dynamic props

Some prop definitions must be computed dynamically, after the user configures another prop. We call these dynamic props, since they are rendered on-the-fly. This technique is used in the Google Sheets Add Single Row action (opens new window), which we'll use as an example below.

First, determine the prop whose selection should render dynamic props. In the Google Sheets example, we ask the user whether their sheet contains a header row. If it does, we display header fields as individual props:

Google Sheets Additional props example - header columns loading as props

To load dynamic props, the header prop must have the reloadProps field set to true:

hasHeaders: {
  type: "string",
  label: "Does the first row of the sheet have headers?",
  description: "If the first row of your document has headers we'll retrieve them to make it easy to enter the value for each column.",
  options: [
    "Yes",
    "No",
  ],
  reloadProps: true,
},

When a user chooses a value for this prop, Pipedream runs the additionalProps component method to render props:

async additionalProps() {
  const sheetId = this.sheetId?.value || this.sheetId;
  const props = {};
  if (this.hasHeaders === "Yes") {
    const { values } = await this.googleSheets.getSpreadsheetValues(sheetId, `${this.sheetName}!1:1`);
    if (!values[0]?.length) {
      throw new ConfigurationError("Cound not find a header row. Please either add headers and click \"Refresh fields\" or adjust the action configuration to continue.");
    }
    for (let i = 0; i < values[0]?.length; i++) {
      props[`col_${i.toString().padStart(4, "0")}`] = {
        type: "string",
        label: values[0][i],
        optional: true,
      };
    }
  } else if (this.hasHeaders === "No") {
    props.myColumnData = {
      type: "string[]",
      label: "Values",
      description: "Provide a value for each cell of the row. Google Sheets accepts strings, numbers and boolean values for each cell. To set a cell to an empty value, pass an empty string.",
    };
  }
  return props;
},

The signature of this function is:

async additionalProps(previousPropDefs)

where previousPropDefs are the full set of props (props merged with the previous additionalProps). When the function is executed, this is bound similar to when the run function is called, where you can access the values of the props as currently configured, and call any methods. The return value of additionalProps will replace any previous call, and that return value will be merged with props to define the final set of props.

# Interface Props

Interface props are infrastructure abstractions provided by the Pipedream platform. They declare how a source is invoked — via HTTP request, run on a schedule, etc. — and therefore define the shape of the events it processes.

Interface Type	Description
Timer	Invoke your source on an interval or based on a cron expression
HTTP	Invoke your source on HTTP requests

# Timer

To use the timer interface, declare a prop whose value is the string $.interface.timer:

Definition

props: {
  myPropName: {
    type: "$.interface.timer",
    default: {},
  },
}

Property	Type	Required?	Description
`type`	`string`	required	Must be set to `$.interface.timer`
`default`	`object`	optional	Define a default interval `{ intervalSeconds: 60, },` Define a default cron expression `{ cron: "0 0 * * *", },`

Usage

Code	Description	Read Scope	Write Scope
`this.myPropName`	Returns the type of interface configured (e.g., `{ type: '$.interface.timer' }`)	`run()` `hooks` `methods`	n/a (interface props may only be modified on component deploy or update via UI, CLI or API)
`event`	Returns an object with the execution timestamp and interface configuration (e.g., `{ "timestamp": 1593937896, "interval_seconds": 3600 }`)	`run(event)`	n/a (interface props may only be modified on source deploy or update via UI, CLI or API)

Example

Following is a basic example of a source that is triggered by a $.interface.timer and has default defined as a cron expression.

export default {
  name: "Cron Example",
  version: "0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "0 0 * * *", // Run job once a day
      },
    },
  },
  async run() {
    console.log("hello world!");
  },
};

Following is an example source that's triggered by a $.interface.timer and has a default interval defined.

export default {
  name: "Interval Example",
  version: "0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: 60 * 60 * 24, // Run job once a day
      },
    },
  },
  async run() {
    console.log("hello world!");
  },
};

# HTTP

To use the HTTP interface, declare a prop whose value is the string $.interface.http:

props: {
  myPropName: {
    type: "$.interface.http",
    customResponse: true, // optional: defaults to false
  },
}

Definition

Property	Type	Required?	Description
`type`	`string`	required	Must be set to `$.interface.http`
`respond`	`method`	required	The HTTP interface exposes a `respond()` method that lets your component issue HTTP responses to the client.

Usage

Code	Description	Read Scope	Write Scope
`this.myPropName`	Returns an object with the unique endpoint URL generated by Pipedream (e.g., `{ endpoint: 'https://abcde.m.pipedream.net' }`)	`run()` `hooks` `methods`	n/a (interface props may only be modified on source deploy or update via UI, CLI or API)
`event`	Returns an object representing the HTTP request (e.g., `{ method: 'POST', path: '/', query: {}, headers: {}, bodyRaw: '', body: {}, }`)	`run(event)`	The shape of `event` corresponds with the the HTTP request you make to the endpoint generated by Pipedream for this interface
`this.myPropName.respond()`	Returns an HTTP response to the client (e.g., `this.http.respond({status: 200})`).	n/a	`run()`

# Responding to HTTP requests

The HTTP interface exposes a respond() method that lets your source issue HTTP responses. You may run this.http.respond() to respond to the client from the run() method of a source. In this case you should also pass the customResponse: true parameter to the prop.

Property	Type	Required?	Description
`status`	`integer`	required	An integer representing the HTTP status code. Return `200` to indicate success. Standard status codes range from `100` - `599`
`headers`	`object`	optional	Return custom key-value pairs in the HTTP response
`body`	`string` `object` `buffer`	optional	Return a custom body in the HTTP response. This can be any string, object, or Buffer.

# HTTP Event Shape

Following is the shape of the event passed to the run() method of your source:

{
  method: 'POST',
  path: '/',
  query: {},
  headers: {},
  bodyRaw: '',
  body:
}

Example

Following is an example source that's triggered by $.interface.http and returns { 'msg': 'hello world!' } in the HTTP response. On deploy, Pipedream will generate a unique URL for this source:

export default {
  name: "HTTP Example",
  version: "0.0.1",
  props: {
    http: {
      type: "$.interface.http",
      customResponse: true,
    },
  },
  async run(event) {
    this.http.respond({
      status: 200,
      body: {
        msg: "hello world!",
      },
      headers: {
        "content-type": "application/json",
      },
    });
    console.log(event);
  },
};

# Service Props

Service	Description
DB	Provides access to a simple, component-specific key-value store to maintain state across executions.

# DB

Definition

props: {
  myPropName: "$.service.db",
}

Usage

Code	Description	Read Scope	Write Scope
`this.myPropName.get('key')`	Method to get a previously set value for a key. Returns `undefined` if a key does not exist.	`run()` `hooks` `methods`	Use the `set()` method to write values
`this.myPropName.set('key', value)`	Method to set a value for a key. Values must be JSON-serializable data.	Use the `get()` method to read values	`run()` `hooks` `methods`

# App Props

App props are normally defined in an app file, separate from individual components. See the components/ directory of the pipedream GitHub repo (opens new window) for example app files.

Definition

props: {
  myPropName: {
    type: "app",
    app: "",
    propDefinitions: {}
    methods: {},
  },
},

Property	Type	Required?	Description
`type`	`string`	required	Value must be `app`
`app`	`string`	required	Value must be set to the name slug for an app registered on Pipedream. App files are programmatically generated for all integrated apps on Pipedream. To find your app's slug, visit the `components` directory of the Pipedream GitHub repo (opens new window), find the app file (the file that ends with `.app.mjs`), and find the `app` property at the root of that module. If you don't see an app listed, please open an issue here (opens new window).
`propDefinitions`	`object`	optional	An object that contains objects with predefined user input props. See the section on User Input Props above to learn about the shapes that can be defined and how to reference in components using the `propDefinition` property
`methods`	`object`	optional	Define app-specific methods. Methods can be referenced within the app object context via `this` (e.g., `this.methodName()`) and within a component via `this.myAppPropName` (e.g., `this.myAppPropName.methodName()`).

Usage

Code	Description	Read Scope	Write Scope
`this.$auth`	Provides access to OAuth tokens and API keys for Pipedream managed auth	App Object: `methods`	n/a
`this.myAppPropName.$auth`	Provides access to OAuth tokens and API keys for Pipedream managed auth	Parent Component: `run()` `hooks` `methods`	n/a
`this.methodName()`	Execute a common method defined for an app within the app definition (e.g., from another method)	App Object: `methods`	n/a
`this.myAppPropName.methodName()`	Execute a common method defined for an app from a component that includes the app as a prop	Parent Component: `run()` `hooks` `methods`	n/a

Note: The specific $auth keys supported for each app will be published in the near future.

# Limits on props

When a user configures a prop with a value, it can hold at most 64KB data. Consider this when accepting large input in these fields (such as a base64 string).

The 64KB limit applies only to static values entered as raw text. In workflows, users can pass expressions (referencing data in a prior step). In that case the prop value is simply the text of the expression, for example , well below the limit. The value of these expressions is evaluated at runtime, and are subject to different limits.

# Methods

You can define helper functions within the methods property of your component. You have access to these functions within the run method, or within other methods.

Methods can be accessed using this.<method-name>. For example, a random method:

methods: {
  random() {
    return Math.random()
  },
}

can be run like so:

const randomNum = this.random();

# Hooks

hooks: {
  async deploy() {},
  async activate() {},
  async deactivate() {},
},

Property	Type	Required?	Description
`deploy`	`method`	optional	Executed each time a component is deployed
`activate`	`method`	optional	Executed each time a component is deployed or updated
`deactivate`	`method`	optional	Executed each time a component is deactivated

# Dedupe Strategies

IMPORTANT: To use a dedupe strategy, you must emit an id as part of the event metadata (dedupe strategies are applied to the submitted id)

Strategy	Description
`unique`	Pipedream maintains a cache of 100 emitted `id` values. Events with `id` values that are not in the cache are emitted, and the `id` value is added to the cache. After 100 events, `id` values are purged from the cache based on the order received (first in, first out). A common use case for this strategy is an RSS feed which typically does not exceed 100 items
`greatest`	Pipedream caches the largest `id` value (must be numeric). Only events with larger `id` values are emitted, and the cache is updated to match the new, largest value..
`last`	Pipedream caches the ID associated with the last emitted event. When new events are emitted, only events after the matching `id` value will be emitted as events. If no `id` values match, then all events will be emitted.

# Run

Each time a component is invoked, its run method is called. Sources are invoked by their interface (for example, via HTTP request). Actions are run when their parent workflow is triggered.

You can reference this within the run method. this refers to the component, and provides access to props, methods, and more.

# Sources

When a source is invoked, the event that triggered the source is passed to run, so that you can access it within the method:

async run(event) {
  console.log(event)
}

# $emit

this.$emit() is a method in scope for the run method of a source

this.$emit(event, {
  id,
  name,
  summary,
  ts,
});

Property	Type	Required?	Description
`event`	JSON serializable data	optional	The data to emit as the event
`id`	`string` or `number`	Required if a dedupe strategy is applied	A value to uniquely identify this event. Common `id` values may be a 3rd party ID, a timestamp, or a data hash
`name`	`string`	optional	The name of the "channel" you'd like to emit the event to. By default, events are emitted to the `default` channel. If you set a different channel here, listening sources or workflows can subscribe to events on this channel, running the source or workflow only on events emitted to that channel.
`summary`	`string`	optional	Define a summary to customize the data displayed in the events list to help differentiate events at a glance
`ts`	`integer`	optional	Accepts an epoch timestamp in milliseconds. If you submit a timestamp, events will automatically be ordered and emitted from oldest to newest. If using the `last` dedupe strategy, the value cached as the `last` event for an execution will correspond to the event with the newest timestamp.

Following is a basic example that emits an event on each component execution.

export default {
  name: "this.$emit() example",
  description: "Deploy and run this component manually via the Pipedream UI",
  async run() {
    this.$emit({ message: "hello world!" });
  },
};

# Logs

You can view logs produced by a source's run method in the Logs section of the Pipedream source UI (opens new window), or using the pd logs CLI command:

pd logs <deployed-component-name>

# Events

If the run method emits events using this.$emit, you can access the events in the EVENTS section of the Pipedream UI for the component, or using the pd events CLI command:

pd events <deployed-component-name>

# Actions

When an action is run in a workflow, Pipedream passes an object with a $ variable that gives you access to special functions, outlined below:

async run({ $ }) {
  // You have access to $ within your action
}

# Returning data from steps

By default, variables declared within an action are scoped to that action. To return data from a step, you have two options: 1) use the return keyword, or 2) use $.export to return a named export from a step.

return

Use return to return data from an action:

async run({ $ }) {
  return "data"
}

When you use return, the exported data will appear at steps.[STEP NAME].$return_value. For example, if you ran the code above in a step named nodejs, you'd reference the returned data using steps.nodejs.$return_value.

$.export

You can also use $.export to return named exports from an action. $.export takes the name of the export as the first argument, and the value to export as the second argument:

async run({ $ }) {
  $.export("name", "value")
}

When your workflow runs, you'll see the named exports appear below your step, with the data you exported. You can reference these exports in other steps using steps.[STEP NAME].[EXPORT NAME].

$.respond

$.respond lets you issue HTTP responses from your workflow. See the full $.respond docs for more information.

async run({ $ }) {
  $.respond({
    status: 200,
    body: "hello, world"
  })
}

return $.flow.exit

return $.flow.exit terminates the entire workflow. It accepts a single argument: a string that tells the workflow why the workflow terminated, which is displayed in the Pipedream UI.

async run({ $ }) {
  return $.flow.exit("reason")
}

$.summary

$.summary is used to surface brief, user-friendly summaries about what happened when an action step succeeds. For example, when adding items to a Spotify playlist (opens new window):

Example implementation:

const data = [1, 2];
const playlistName = "Cool jams";
$.export(
  "$summary",
  `Successfully added ${data.length} ${
    data.length == 1 ? "item" : "items"
  } to "${playlistName}"`
);

$.send

$.send allows you to send data to Pipedream destinations.

$.send.http

See the HTTP destination docs.

$.send.email

See the Email destination docs.

$.send.s3

See the S3 destination docs.

$.send.emit

See the Emit destination docs.

$.send.sse

See the SSE destination docs.

# Environment variables

Environment variables are not accessible within sources or actions directly. Since components can be used by anyone, you cannot guarantee that a user will have a specific variable set in their environment.

In sources, you can use secret props to reference sensitive data.

In actions, you'll see a list of your environment variables in the object explorer when selecting a variable to pass to a step:

# Using npm packages

To use an npm package in a component, just require it. There is no package.json or npm install required.

import axios from "axios";

When you deploy a component, Pipedream downloads the latest versions of these packages and bundles them with your deployment.

Some packages — for example, packages like Puppeteer (opens new window), which includes large dependencies like Chromium — may not work on Pipedream. Please reach out (opens new window) if you encounter a specific issue.

# Referencing a specific version of a package

This currently applies only to sources.

If you'd like to use a specific version of a package in a source, you can add that version in the require string, for example: require("axios@0.19.2"). Moreover, you can pass the same version specifiers that npm and other tools allow to specify allowed semantic version (opens new window) upgrades. For example:

To allow for future patch version upgrades, use require("axios@~0.20.0")
To allow for patch and minor version upgrades, use require("axios@^0.20.0")

# Managing Components

Sources and actions are developed and deployed in different ways, given the different functions they serve in the product.

Managing Sources
Managing Actions

# Managing Sources

# CLI - Development Mode

The easiest way to develop and test sources is with the pd dev command. pd dev deploys a local file, attaches it to a component, and automatically updates the component on each local save. To deploy a new component with pd dev, run:

pd dev <filename>

To attach to an existing deployed component, run:

pd dev --dc <existing-deployed-component-id> <file-or-name>

# CLI - Deploy

# From Local Code

To deploy a source via CLI, use the pd deploy command.

pd deploy <filename>

E.g.,

pd deploy my-source.js

# From Pipedream Github Repo

You can explore the components available to deploy in Pipedream's GitHub repo (opens new window).

pd deploy <source-key>

E.g.,

pd deploy http-new-requests

# From Any URL

pd deploy <url-to-raw-code>

E.g.,

pd deploy https://raw.githubusercontent.com/PipedreamHQ/pipedream/master/components/http/sources/new-requests/new-requests.js

# CLI - Update

View the CLI command reference.

# CLI - Delete

View the CLI command reference.

# UI - Deploy

You can find and deploy curated components at https://pipedream.com/sources/new (opens new window), or you can deploy code via the UI using following URL patterns.

# From Pipedream Github Repo

https://pipedream.com/sources?action=create&key=<source-key>

E.g.,

https://pipedream.com/sources?action=create&key=http-new-requests

# From Any URL

https://pipedream.com/sources?action=create&url=<url-encoded-url>

E.g.,

https://pipedream.com/sources?action=create&url=https%3A%2F%2Fraw.githubusercontent.com%2FPipedreamHQ%2Fpipedream%2Fmaster%2Fcomponents%2Fhttp%2Fhttp.js

# UI - Update

You can update the code and props for a component from the Configuration tab for a source in the Pipedream UI.

# UI - Delete

You can delete a component via the UI at https://pipedream.com/sources (opens new window).

# API

See the REST API docs).

# Managing Actions

# CLI - Publish

To publish an action, use the pd publish command.

pd publish <filename>

E.g.,

pd publish my-action.js

# Source Lifecycle

# Lifecycle hooks

Pipedream sources support the following hooks. The code for these hooks are defined within the component. Learn more about the component structure and hook usage.

# `deploy`

The deploy() hook is automatically invoked by Pipedream when a source is deployed. A common use case for the deploy hook is to create webhook subscriptions when the source is deployed, but you can run any Node.js code within the deploy hook. To learn more about the deploy() hook, refer to the API documentation.

# `activate`

The activate() hook is automatically invoked by Pipedream when a source is deployed or updated. For example, this hook will be run when users update component props, so you can run code here that handles those changes. To learn more about defining a custom activate() hook, refer to the API documentation.

# `deactivate`

The deactivate() hook is automatically invoked by Pipedream when a source is updated or deleted. A common use case for the deactivate hook is to automatically delete a webhook subscription when a component is deleted, but you can run any Node.js code within the deactivate hook. To learn more about the deactivate() hook, refer to the API documentation.

# States

# Saved Component

A saved component is non-instantiated component code that has previously been deployed to Pipedream. Each saved component has a unique saved component ID. Saved components cannot be invoked directly — they must first be deployed.

# Deployed Component

A deployed component is an instance of a saved component that can be invoked. Deployed components can be active or inactive. On deploy, Pipedream instantiates a saved component and invokes the activate() hook.

# Deleted Component

On delete, Pipedream invokes the deactivate() hook and then deletes the deployed component instance.

# Operations

# Deploy

On deploy, Pipedream creates an instance of a saved component and invokes the optional deploy() and activate() hooks. A unique deployed component ID is generated for the component.

You can deploy a component via the CLI, UI or API.

# Update

On update, Pipedream, invokes the optional deactivate() hook, updates the code and props for a deployed component, and then invokes the optional activate() hook. The deployed component ID is not changed by an update operation.

# Delete

On delete, Pipedream invokes the optional deactivate() hook and deletes the component instance.

# Source Event Lifecycle

The event lifecycle applies to deployed sources. Learn about the source lifecycle.

# Diagram

Pipedream Components Event Lifecycle Diagram

# Triggering Sources

Sources are triggered when you manually run them (e.g., via the RUN NOW button in the UI) or when one of their interfaces is triggered. Pipedream sources currently support HTTP and Timer interfaces.

When a source is triggered, the run() method of the component is executed. Standard output and errors are surfaced in the Logs tab.

# Emitting Events from Sources

Sources can emit events via this.$emit(). If you define a dedupe strategy for a source, Pipedream automatically dedupes the events you emit.

TIP: if you want to use a dedupe strategy, be sure to pass an id for each event. Pipedream uses this value for deduping purposes.

# Consuming Events from Sources

Pipedream makes it easy to consume events via:

The UI
Workflows
APIs
CLI

# UI

When you navigate to your source in the UI (opens new window), you'll be able to select and inspect the most recent 100 events (i.e., an event bin). For example, if you send requests to a simple HTTP source, you will be able to inspect the events (i.e., a request bin).

# Workflows

Trigger hosted Node.js workflows on each event. Integrate with 1,400+ apps including Google Sheets, Discord, Slack, AWS, and more!

# API

Events can be retrieved using the REST API or SSE stream tied to your component. This makes it easy to retrieve data processed by your component from another app. Typically, you'll want to use the REST API to retrieve events in batch, and connect to the SSE stream to process them in real time.

# CLI

Use the pd events command to retrieve the last 10 events via the CLI:

pd events -n 10 <source-name>

Installing the CLI →

# Component API Reference

# Overview

# What is a component?

# Quickstarts

# Differences between sources and actions

# Getting Started with the CLI

# Example Components

# Component API

# Component Structure

# Props

# User Input Props

# General

# Advanced Configuration

# Async Options (example (opens new window))

# Prop Definitions (example (opens new window))

# Referencing values from previous props

# Dynamic props

# Interface Props

# Timer

# HTTP

# Responding to HTTP requests

# HTTP Event Shape

# Service Props

# DB

# App Props

# Limits on props

# Methods

# Hooks

# Dedupe Strategies

# Run

# Sources

# $emit

# Logs

# Events

# Actions

# Returning data from steps

# Environment variables

# Using npm packages

# Referencing a specific version of a package

# Managing Components

# Managing Sources

# CLI - Development Mode

# CLI - Deploy

# From Local Code

# From Pipedream Github Repo

# From Any URL

# CLI - Update

# CLI - Delete

# UI - Deploy

# From Pipedream Github Repo

# From Any URL

# UI - Update

# UI - Delete

# API

# Managing Actions

# CLI - Publish

# Source Lifecycle

# Lifecycle hooks

# deploy

# activate

# deactivate

# States

# Saved Component

# Deployed Component

# Deleted Component

# Operations

# Deploy

# Update

# Delete

# Source Event Lifecycle

# Diagram

# Triggering Sources

# Emitting Events from Sources

# Consuming Events from Sources

# UI

# Workflows

# API

# CLI

# `deploy`

# `activate`

# `deactivate`