# Deploying LLMs with Streaming Responses

Eli Mernit

December 3, 20241 min read

> **_This is the second feature in our [launch week](https://x.com/beam_cloud/status/1862543368217141381), where we ship a brand new feature every day, five days in a row._**

## **Background**

Today we're officially releasing a realtime API to stream websocket responses from your Beam apps.

Technically, it's been possible to stream responses for a while. However previously, users had to write websocket logic into their Beam apps manually using FastAPI.

This wasn't terrible, but it introduced a lot of boilerplate into users' application code.

We wanted to hide this ugly websocket logic from users, so we decided to add a `@realtime` decorator which automatically creates and manages websocket connections from a Beam app.

## Using Streaming Responses

Here's how to deploy a realtime endpoint on Beam. Under the hood, this is a wrapper on top of our [ASGI](https://github.com/beam-cloud/beta9/blob/b114dc6e715d1365dc249e46f8a6f81cbdbb35b1/sdk/src/beta9/abstractions/endpoint.py#L160) decorator. All the features in ASGI apps -- such as [concurrent requests](https://docs.beam.cloud/v2/endpoint/web-server#concurrent-requests), [callback URLs](https://docs.beam.cloud/v2/topics/callbacks#task-callbacks), and [custom base images](https://docs.beam.cloud/v2/environment/custom-images) \- can be tacked onto your realtime apps.

```python
from beam import realtime

@realtime(
    cpu=1,
    memory="1Gi",
    concurrent_requests=10, # Process 10 requests at a time
    authorized=False, # Don't require auth to invoke
)
def stream(event):
    # Echo back the event payload sent to the websocket
    return {"response": event}
```

Connecting to a realtime app from your client is straightforward:

```javascript
import beam from "@beamcloud/beam-js";

const streamResponse = async () => {
  const client = await beam.init(process.env.BEAM_TOKEN);
  const deployment = await client.deployments.get({ id: process.env.BEAM_DEPLOYMENT_ID });

const connection = await deployment.realtime();

const payload = {
    "event": "Echo this back",
  }

connection.onmessage = (message) => {
      console.log(`🎉 Response: ${message.data}`);
  };

connection.send(JSON.stringify(payload));

setTimeout(() => {
    connection.close();
  }, 1000);
};

streamResponse();
```

## Running Concurrent Requests

You can specify the number of concurrent requests your app can handle using the `concurrent_requests` parameter in the `@realtime` decorator.

```python
@realtime(
    image=Image(python_packages=["transformers", "torch"]),
    memory=1024,
    concurrent_requests=10
)
```

This allows you to increase the number of requests your app can handle at once, which can help you achieve higher throughput. For instance, if your app is doing I/O-bound work, additional requests can be handled while your I/O operations complete in the background.

## Get Started Today

To get started, [create a Beam account](https://platform.beam.cloud/) and [follow this guide in our docs](https://docs.beam.cloud/v2/endpoint/realtime). If you have any feedback on the workflow or feature requests, we’d love to hear from you in our [Slack Community](https://join.slack.com/t/beam-cloud/shared_invite/zt-2uiks0hc6-UbBD97oZjz8_YnjQ2P7BEQ).

> **This is Launch 2/5 this week! You can follow along with our upcoming launches on [Twitter](https://x.com/beam_cloud/status/1862543368217141381).**