Build Advanced Content Workflows: A Deep Dive with Directus + Inngest

Speaker 0: With Directus and Inngest. I'm your host on the Directus side, Bryant Gillespie, developer advocate, growth engineer, professional hack and slash developer over here at Directus. Super excited for today. Got a special guest that I'll introduce in just a moment. We should have a good time.

We kinda reworked the slide deck a bit last minute, as I tend to do. But before we dive into introductions, let's just take a look at what's on the agenda. Alright. So we'll go through some super awkward introductions in just a moment. We'll deep dive into what is ingest, what powers ingest behind the scenes, why you would wanna use ingest, then we'll get into why Directus and ingest are a great pair together.

And then we're gonna run through this advanced content workflow after we've, walked through this integration of invincible automatic localization, for your content, which is always a beast. So with that, I want to kick over to mister Dan. Dan, how are you, sir?

Speaker 1: Great. Great, Brian. Thanks for thanks for having me. Stoked about stoked about doing this. Thanks everyone for joining as well.

My name is Dan, and I'm the CTO and cofounder at a company called Jest. And we'll, we'll get into a little bit more about what that is and, what we're doing today and, why you've joined. But, yeah, just thanks for joining, and I'm stoked about the stuff that Brian, Brian kicked off today.

Speaker 0: Yeah. Yeah. And, Abel, like, for for some context and backstory, like, I first found ingest, we were working on a project we were calling Event OS. And, Direct has made it a breeze to set this up, get some APIs to work with inside a Nuxt application, but it's not dissimilar from the platform you use to register for this event. One of the the main challenges was, like, how do we send notifications and, like, how do we set up reminders an hour before an event and still have that be robust, like, cancellable in case we move the event, you know, and all sorts of things that we're gonna cover today.

So that was my introduction to ingest, and frankly, I was kinda blown away. I I don't think if Matt is here in the chat, I was like, man, we gotta get these guys on a partner webinar, and voila, here we are. So with that, we'll just drag the little slide deck over. And what's up next is what is ingest? So I'll kick it over to you, Dan, and, you know, let you dive into ingest a little bit deeper for those who are unfamiliar.

Speaker 1: Great. Thanks, Brian. Appreciate it. So I'm gonna give you a little high level overview today, and we're a technical audience, so have a little bit more of a a technical variant on slides today. So, if anyone needs things a little bit bigger, tell me to bump it up in the chat.

I'll try to respond if there's anything. So yeah. So, basically, what is Ingest? Okay. What you can basically think about it as the at the high level is that Ingest is a workflow orchestration platform that enables you to run reliable asynchronous jobs anywhere.

We're gonna get come back to that in just a just a minute or two. So why would you use ingest for this this task, this need? What ingest allows you to do is to define these step functions. So you're creating workflows directly in your existing code base, use our SDKs, and any logic that you're used to. There's no DAGs or really obscure complex DSLs to learn to build these, powerful functions using regular code.

So they're also reliable by default. So everything that you run with ingest automatically is retries. I think we've probably a lot of people here have built systems where, things go wrong. An API goes down for a few minutes, there's a blip, there's a networking thing, there's a race condition that happens in your database, and all simple retry a few seconds later would solve your problem. So ingest does that by default, and it also includes, something that is called durable execution.

You may have heard this term, and if you haven't, you can think of this as your code is designed to be fault tolerant and survive catastrophic failovers, basically. So I won't get too deep into that topic today, but there's plenty of resources that we can send you after if you are curious. The last thing is that we've built in queuing and flow control right into ingest so that you can go to production confidently. What that means is that there are concurrency management built into the system that you don't need to have run a a worker or a queue or anything like that. You just define it in your code very simply.

You also have controls for multi tenant support, which is another advanced topic we won't get into yet today. But you can just kind of anything that you might have built into some sort of, asynchronous system that you might need is there at your fingertips with easy configuration like throttling, rate limiting, delaying the start of jobs, debouncing, batch processing, dynamically prioritize prioritizing certain jobs, as well as running work in parallel or fanning out to multiple jobs. So that's at the high level, but realistically, okay, what can I build with ingest? So let's just you know, at the highest level in general thing is ingest is a general purpose tool, and you can run any background task. Something simple, just one function, one operation to something that is that is very complex and has to orchestrate between different process.

So you also have the control in your hands. You can control the flow of the logic with if statements, not DAGs, and you can kinda build whatever you want. So I'll give you a little taste of, you know, with this audience, which is technical, and I'm sure a lot of y'all are interested or already users of Directus. You know, what might work in or make sense in your in your situation or what you might be building? So one of the first things that's very hot, sure a lot of you are dabbling or building already, pushing these things to production is AI workflows.

So you can handle the complexity of things like flaky LOM API calls that might fail, things like chaining, building agent loops, calling tools if you're getting that advanced, or do things like human in the loop flows. You know, you're you're proving the work that that, LLM did for you. So this might be a research AI workflow. Go out and get all this data and pull it back and store it somewhere, or you might be creating content. That kinda leads me into content processing, which is where you might be pre or post processing some content.

It might be a blog post or could be static assets that are in your system like images or videos. You might be doing things like translating, resizing, transcribing more to those assets. You might need to kind of create a complex pipeline. So one example here is that you you might be taking data, from your, your Directus system and you might be piping that into maybe a React SVG library and generating visual assets on the fly for your content and then uploading it to where you're storing, your assets. You know, another thing there, like, kind of beyond content is is user journey automation.

Brian said at the top something about scheduling emails with events and and and and that and and whatnot. Within just, sorry, Brian.

Speaker 0: No. No. That was a huge one for me, man. That was, that was, like, my introduction to it, and I think that was, like, one of the one of the first case studies or something I ran across in the docs of, like, hey. Just something as simple as let me sign a user up and then wait a day and send them something Yeah.

Incredibly convoluted in other systems.

Speaker 1: Yeah. And and you know what? You might have this thing where you could some systems might allow you to schedule an email in the future, or you might have a cron based system. But what if you could just, you know, prepare something, do some operation maybe like you're sending somebody you know, you're sending someone an invite, and then that invite email goes out. And then twenty four hours before the event, you might wanna schedule and say, let's render this email.

Might be using, say, React email, which is awesome, and, pulling in data from different sources and sending that person a personalized email twenty four hours ahead. Or you might wanna customize, hey. This user is in this time stamp. So you might wanna, like, have more control sorry, time, time stamp, time zone. So you might wanna have, like, a time aware thing.

So you could use something like another complex tool or, whatever is within an event platform that you're doing or something like Intercom or HubSpot, but that can be either not very controllable. You can't bring in custom logic and code. It's very static. They're crazy expensive, for those purposes. So you kinda do whatever you want here.

Build flows that are scheduling delays, Slack notification, even like things like provisioning resources or provisioning access. So, that's a fun kinda use case. You can kinda build anything triggered with these vents. I'll go through a couple of these other ones because these are kinda secondary. But, you know, if you're integrating your system with a third party webhook, maybe something like integrating Stripe payments somehow into in into your system, or maybe you're using a system like Mux, and you're uploading your video assets to Mux, but you need to do something in your Directus CMS after those videos are uploaded, you can use a webhook.

And you can those events can go to ingest and trigger your functions to run your Directus instance. So you can kind of build more advanced integrations with more control that you might need. And the last thing that happens in a lot of systems is there's always data from one source to another to another place. So you might have just data import jobs. Kind of think about it like you're building ETL jobs or just pipelines or synchronization, but you don't, you know, you you don't need anything like a complex data science tool or, something that a data engineer might use like an airflow or something like that.

So you can build kind of anything in this, but I hope these are kind of ideas that might work with, what you're already doing with Directus. So I'll go from there, and I'll, I'll get to that question in a little bit. We have some time at the end as well. And, I'll go and just show, like, a little bit of functions. You this again is a technical audience, so let's just show some code about how you build an Ingest function and how it works.

So I'll run through this. Basically, what you're doing here is you're creating the Ingest function. It's pretty simple. You define it in your existing code base along with your directus functions, your directus logic and whatnot, your whole system. You can do things like define queuing or flow control right at the top of the function very easily.

So if, for example, you just wanted to run this job, at most 30 times in sixty seconds and you wanted to max out at three retries or 15 retries, you can control these things very easily and the ingest system will just do it automatically for you. You don't need to battle queues and infrastructure for that. You also declaratively define the event that will trigger your function. So in this case, when a blog post is drafted, run this code. And then bay very simply, you define a function right here, which is a handler that has all your code.

If you're thinking about what I talked about before which is workflows, steps, and whatnot, when you're defining ingest functions, all you need to do is learn a couple building blocks which are steps. The main one is step dot run. Basically, you can encapsulate some sort of business logic, some sort of side effect, and you can perform an operation in there. Each time you run code within one of these, it's automatically retried if it fails as per your retry policy. And if this does succeed, if there's any failures later in the function, it won't be retried.

We cache those results and save them. So it makes your function a little durable if you're familiar with that term or fault tolerant, you know, that's the the a la the, durable execution side of things. So, basically what you're doing is you're defining steps and multiple steps to build out your workflow. You can use normal if logic. You can dynamically define steps on the fly.

You do not need to predefine them, which is really great. And that's the building blocks. But what's interesting is that there's more power to these these these basic primitives of steps that you have. So here's one example that might tie back to what we were just talking about, which are that you can pause workflows and wait for approval. So you can or wait for other data to happen, other events.

So you could build things that run, process some, you know, some data, and then wait for maybe an approval or maybe something else in your system to be updated, like a draft or publish status or something, and you can resume your function. So ingest your code will not be running. Ingest basically holds on to the state of your function. And then when it gets that event, it resumes and restarts your function from where it stopped, doing some different kind of magic and whatnot. So then again, you can use that approval to dynamically, you know, traverse and and run different steps.

You can also do things that are a little bit, you know, kind of outside of this this this thing that are that are kind of extending the power, which are making reliable calls to LLMs by offloading those AI calls to ingest servers. You can also do things like what Brian talked about, which are basically use other types of steps like sleeps or delays. So you can delay to an exact time and then resume the code then, or you can delay to an arbitrary number of days or hours or weeks, and then your code resumes. So these are kinda just the the basic building blocks. There's a couple more that I didn't get into.

But if you can learn these couple primitives, you can build a lot of complex logic that also is very reliable outside of box. So, you know, going from here, I'll just take a step back and look at the architecture. And, Brian, how are we doing on time right now?

Speaker 0: We are we're no. We're golden.

Speaker 1: Can you see? Okay.

Speaker 0: Yeah. Yeah. No. I just yeah. No.

We're beautiful. No worries. I I I did just wanna say, like, the hey, like, the the background jobs and and, like, some of the the queuing stuff was what got me in the door. But, like, the the durable execution piece, especially when you're working with, like, the LLM stuff, after you've you've used, like, one of these big models where you could just literally watch credits evaporate when a function fails is, has been a, like, a nice piece for my workflow. So, kudos there.

Speaker 1: Yeah. Thank you. Thank you. Appreciate that. And, yes, to the to the, to the audience, yes.

Using, step dot a I dot infer, if you're running in serverless because with something like Vercel, it can offload it to our servers. So your function stops running. It's not consuming any any compute and it offloads. So that was an easy one to jump in there. Thanks for that question.

Yeah. I'll jump into the architecture because I wanna show you how it works with Directus and how the system is. So basically, at at the high level, this is a little, you know, you don't need to know, like, know everything in detail. I'll try to zoom and see if I can pinch in a little bit. There we go.

So the ingest system comprises a bunch of things that you might typically build or need to to execute this and do this yourself. So and there's much more beyond that. But the fundamental thing is there's an event API that receives data from anywhere, could be from your direct assistance or the webhook, And there's, something that consumes those events and basically, creates new jobs and enqueues them. It stores the state of those jobs. It stores the history of those jobs in case there's a problem or anything or you wanna observe that later, which we'll get to, then it executes those jobs.

And what happens is the ingest knows, okay, I'm running this function and it toasted on your direct instance. Your code is sitting in your direct instance, and it invokes it via HTTP. So it securely just calls it when it needs to, invokes the minimum number of work, and then returns to ingest. And then if it needs to resume or continue, it will make other calls as necessary. So if you sleep, it goes back to ingest, then it calls your server again later.

So this is the basic kind of principle of how all the systems work together and how there's a queue built into the system that is basically pushing work to your system. And that's why it's really key to have the flow controls. You know, you don't want to you wanna set something like throttling because you don't wanna overwhelm and knock down your system or your database in case you're processing, say, a huge backlog of 10,000 or 10,000,000, jobs or tasks that you need to do. So this is really key and also just shows that the code that you write is hosted on your, your machine. It's where you have it.

You keep it along with your existing, direct as code base, which is really great. So I'll, I'll show you a little bit about, like, why do we have a database, what is this dashboard UI. So when you're running these long jobs that are happening behind the scenes, they're often hard to observe. I'm sure that some of you have worked in systems where you're parsing just logs on CloudWatch or something like that. You're trying to figure out what's the status of this job.

Okay. Now we need to chuck it into a database so we can observe this. And you're trying to piece together what happened and where things failed. So we get to that one question in one second. So what we have is we have a dashboard that gives you visibility into the system.

So this is the ingest dashboard and ingest cloud where you can see the the output or status of all the different workflows and functions that you've run-in your system, and then you can expand these to see all the different steps and what might have failed, what have might retried, the outputs of those steps, some metadata that you might pull through like, say, tokens in the model that you might have used to make an AI call or the time stamp when something started or ended. You also can look at inputs, say, okay. What piece of content triggered this? You can debug what actually happened and understand, did this work? You can easily cancel and retry these things or retry them in bulk.

There's a lot of operations you could do, but this is the core of, like, this observability that you have into the system. So in also at the, like, the highest level, you can see broad system level metrics, like what functions have failed. Is my import data pipeline failing at a high rate? What what's happening? I need to look into this.

So it gives you this kind of overall picture of how the system is go is going where it's it might be doing things always in the background. It's it aren't always automatically visible. And you can run many functions and have all different types of, functions and workflows like generating video transcripts or creating chat completions or daily digest. You you just get the single pane of glass of, like, understanding what's running in my system. And so the point of concurrent runs that, was asked in the, in the, in the chat, what happens is basically, like, ingest, can can execute multiple things at a given time.

Depending on your plan and your configuration, you can limit how much concurrency you want. So you can control I only wanna execute 1,000 things at a time, or you might have multiple servers that you've horizontally scaled to handle more load. You can control individually what that is. You can also with the multi tenant controls can can control how many concurrent jobs a given user gets. So you can limit it like that in case you're building that type of system.

But, ingest handles that because it it it does the it actually processes the the work off the queue internally. It can you basically declaratively say, this is the concurrent number of like, amount of work that I wanna run. So that's a that's a great question though. So, you know, basically, to summarize, like, this part, there's why are we here? We're talking about in ingest and direct us.

Right? So tie back a couple things that I that I went over. It's the same code base that you're already running. It's the same server. You get to take right.

Use the ingest SDK to define the functions right in the same code base with everything else that you're using with Directus, and there's no additional compute necessary. And, and, Brian, you got some? No.

Speaker 0: No. I was just jumping back in as we transition.

Speaker 1: Yeah. I'll, I'll I'll, Brian's gonna go in a lot deeper about this, which I'm which I'm stoked for all y'all to see because he's built some really cool things. And and some of the things again, this is fault tolerance, the async workflows, you know, so you get the confidence, automatic retries, and control over these processes. And remember that steps encapsulate these retryable logic. You can bring existing logic, import new libraries, whatever the heck you want, write any workflow.

And one thing that Brian will go deep on this, I think, is super cool about how it works together is Directus, hooks are super cool because it kinda provides this nice link between the direct to system and ingest functions, but I won't spoil anything there. I'll, I'll pass over to Brian to kinda take it from here. But, and then we can go to if there's any other questions, we'd probably go to the the QA afterwards so we can get through some of these things.

Speaker 0: Some of

Speaker 1: the questions might be answered.

Speaker 0: Yeah. Definitely. So I pop the questions into the chat. We'll get to them if we can. I'll take the screen back, and we'll briefly cover what is Directus.

For those, who are joining, who are not familiar, Directus is I I like to call it LEGO for developers where, like, everything that you need to build a digital experience, whether it's an app, a website, automatic content translation, which we're gonna do today, it it's all there for you. And you're just stacking these bricks together a lot like, what Dan was mentioning with ingest of, you know, these primitives that you have, and you could use them to build really powerful stuff. So Directus will give you intuitive UI, which we'll look at in a moment, a nice, like, admin CMS layer. You get instant rest and GraphQL APIs on top of any SQL database, and this is all self hostable. You can also run-in our cloud as well.

And just a little tiny, fine print there of any, should be like most SQL databases. Right? Alright. So why Directus? Why ingest?

Why together? A couple things. Like, Dan already mentioned, you know, the hooks that we have in Directus make this super easy to communicate with ingest. But more than that, ingest gives you that durable execution. To say being able to build these powerful workflows in code is tremendous.

So the first question that, I was asked when I was putting this webinar together, and and this is from our team was, hey. Why why ingest when we already have automations inside Directus? So if you're still learning Directus, we have an automations module called Directus flows, and it is allows you to define a low code, no code automations. You know, when something happens, do this. These flows are great for short lived automations.

Some of the things that that Dan showed in the previous slide where we have the syntax of, like, the step function, Directus doesn't have the ability inside a flow to wait for thirty days and send an email or to, wait for another event inside of a flow, without some complicated logic tying it all together. So flows are great for short lived automations or automations that that don't have super complex logic, and ingest is a great complement to flows. And, you know, in the future, I could even see us having an extension to trigger ingest, functions inside a flow. But, as we get into flows versus hooks versus extensions, as as Dan mentioned, Directus flows underneath the hood once you pull off the the mask here, the the Scooby Doo, I kinda mean, is just using Directus hooks. So, Directus hooks, if we pull up my browser, which is always dangerous since I'm using Arc, Directus hooks are just code that fires whenever certain events occur, and you've got a lot of different hooks at your disposal.

This is using custom extensions, and we'll kinda cover that in just a minute. But at the core, hooks are allowing you to say, hey. When this event happens at Directus, do this. And, we'll we're leveraging these hooks extensively in this integration. So there's filter hooks, which happen before the event is emitted.

So if I wanna run some code to either modify the payload before an item gets created or potentially stop that from occurring, I could do that via a filter hook. And then the ones that we're leveraging today in this workflow are gonna be our action hooks. So when after an item gets created, we wanna do something. In this case, it's going to be sending those events to ingest. Alright.

Any questions on hooks before we kinda dive into this? We've also got, like, an endpoint that we're gonna use, and I I just wanna take a moment to touch on one of the most important pieces of Directus is this extensibility. So if you go into our documentation, which is a a great starting point after this webinar, take a look at the extensions overview. You could customize every piece of Directus. So you could customize the interfaces, like how the data is displayed inside the studio.

But the parts that we're leveraging today are API extensions. The hooks, endpoints, operations are within flows, and, of course, bundles are how we put all these together and distribute them. Alright. So how do we make Directus and Ingest play nice? And this was the part that, I did the heavy lifting on with some help from Dan and his team.

You're still blown away by how easy it is to write ingest functions. It really just kinda gets out of the way. And and once you guys go through this process and set this up and and don't worry. Wait. There's more type of moment here.

We're we're gonna give you the whole code base as the bonus after this, so don't worry about taking notes or screenshots or anything like that. But, once you've got this set up, then you can build really incredible stuff, with the incredible pace, I would say. Alright, so I'm going to open up my code editor here and we're gonna take a look at the first piece of the puzzle which is our environment setup. And for Directus projects this almost always starts with a Docker composed file. So this is the pretty standard boilerplate Docker composed from, our documentation.

I've I've done just a little bit of cleanup here and extracted some of these things out to environment variables versus, just inlining them. And a couple things that I want to note, when you spin up Directus, especially, like, using our standard Docker Compose, which we're using Postgres for the database, Directus will create this, this folder structure for you with the database, the extensions. This template is is just for applying the schema that we've got for this example, but, it will create all of that for you. And for a cleaner DX, I've basically just extracted this custom Directus extension out to the root in a queue folder. So this entire code base, I've I've gone through the trouble of trying to make comments, to to add a little bit of what I was thinking as we went through here.

But all the the rest of this is pretty standard with the exception of a different port. I'm sure Dan can relate to this as you're developing. You've got, like, 35 instances of your product running on your local machine at one given time, so you gotta switch these ports over. The only addition here is the ingest dev server. So the ingest dev server, they've got a a docker image that you could pull.

You can also run this via NPM too. Right? Dan, a node?

Speaker 1: Yeah. Yeah. You can. You can install the binary via MPX or NPM.

Speaker 0: Perfect. Perfect. Yeah. So, you know, we highly recommend Docker if you're working with Directus, but, you know, you could run this, ingest dev server using NPM if you prefer. But I just wanted to point this out like we've got it pointing to our direct assistance, and this is running on port eight two eight eight.

The dev server is amazing for running locally. It's almost, identical to the ingest cloud dashboard, with a few exceptions. And, you know, one thing I do wanna call out, and I think Dan already iterated on it, is your ingest functions, unless you're doing, like, the the LLM infer that that Dan mentioned, these are all running on your same Directus instance whether you're using the dev server or ingest cloud. So I just wanted to call that piece out. Now for the environment that we've got set up here, you could see when you go to production and just has event signing keys or event keys and signing keys so you could keep, things very secure.

For purposes here, we're just running, on the dev mode, keep it lightweight, make it easy to iterate. One thing to note, when we get into the translation workflow, we are using DeepL, so I've just got that environment variable. So when we are configuring a Directus extension, Directus has a extensions SDK that makes this process super easy. I just bang in this command into my terminal, n p x create direct us extension. Make sure you use the at latest tag to to pull that.

And what you're going to initialize, I'll just show you kinda what this looks like, is a bundle extension. MPX Directus create extension at latest. This will ask you to install, and then we will go through and choose a bundle. And this is where my Internet just totally craps the bed apparently. Too many connections at once.

Okay. Yep. So we scroll down, we hit a bundle, and a bundle is just a collection of these extensions. Now once you initialize that, it will look something like this. Where are we?

So I've I've got this in our queue set up. You could see here we've got our Directus extension. It is a type of bundle, and then to add extensions to that bundle you run npm add or npm run add or p npm add, whichever package manager that you're using. So as far as the ingest setup here, let's dive into what that actually looks like. And, again, we're gonna give you access to all of this code.

So at the high level here, we've got a function folder that stores all of our functions. We've got some hooks that we'll take a look at, and then we have this ingest client. So how do we set this up? How do we make sure it connects securely and plays nice with Directus? We are pulling in from the ingest package, and we're gonna skip this little middleware piece, but this create ingest client basically just returning a new ingest client that we can use to invoke these functions.

Now the middleware piece here is rather important. Basically, all we're doing here is some dependency injection to use direct to services, like creating an item, updating an item, sending a notification to our user. So that's an important piece of the puzzle. There's a a couple things like a helper function in here to set the directest context so we can leverage that. And then we're just basically using a singleton pattern for this ingest client so that, you know, we're we're using the same client over and over.

Setting this up and, you know, ingest needs to be able to serve those functions. So, in that we've got a endpoint that we define. So this is a custom direct us endpoint. You know, you can see some some standard, imports here. This is where we're actually going to import those functions.

We'll show that in in just a few, but we're basically defining an endpoint. The ID here is gonna be the route that this gets served on. So, I'm using ingest. If I go to local host eight zero eight eight slash ingest, that's where my functions will be served from. And you can see here when we're defining an endpoint, we get the express router instance, and then we get the direct us context.

And that is what we are calling here or or what we're passing when we set the directest context so that ingest has access to those directest services. Very important piece. If you're setting this up on your own, don't, don't forget that part of it. Last but not least, we'll basically just serve the ingest client here. You can see the syntax we're passing in the client that we've set up, and then we're giving it the functions to serve.

Now the last piece of making these actually talk to each other is the direct us hooks. Right? So, we've got our hooks set up. The famous handler. I see.

Not sure what you mean by that, Bazar, but, I we can unpack that in the q and a for sure. Alright. So hooks. Right? Again, we're just defining these actions that we want to run whenever a certain event occurs.

And I was I was using I I won't say I was using this wrong, but Dan definitely gave me a level up, while we were prepping for this webinar. So, breaking these down. Right? We're using the action hooks inside Directus. So I only want to invoke a ingest function after a certain thing occurs, not not necessarily before.

So if we break this down, whenever a post so we'll get into our data model in a moment, but whenever our post gets updated, we are going to send, ingest an event that our posts were updated and we're gonna pass the the event from Directus and this accountability object from our context, which is basically this is the user permissions and, like, the user ID and things like that. Is this user going to have access to actually update that post, after we do the translations that we take a look at. So

Speaker 1: one

Speaker 0: of the things that I I do wanna recommend as you go through this, mirror the event names inside your ingest functions from your direct us, events. So you could see this syntax here. Basically, I've just added direct us in case we've got some other service that we want to send events to ingest from. But here, I've just mirrored the syntax and the reason why, and Dan schooled me on this, is it makes it easier to debug and trace these things through your application, especially if you want to trigger multiple ingest functions on a post update or when a post gets created. Alright.

So the last piece of our puzzle. Right? We've added hooks. We will create our ingest functions, and we'll take a look at that. We'll share this amazing slide deck at the end of this as well.

But we've got a full guide if you've already got a direct us project and you just wanna walk through integrating this. We've got a full guide that we've written on our documentation for you to check out as well. Alright. So, Dan, I'm gonna bring you back. This was your diagram.

You know, maybe maybe kinda touch on how this works a bit.

Speaker 1: Yeah. Yeah. So I think, Brent really laid it out really well here in that. He just showed where with the extension and direct us hooks, that's where we're we're hooking to these actions and just broadcasting basically events over to ingest. So So those events, are basically declaring what happened within the direct to simple, system.

And then our ingest functions, will declare when they run. Right? We we saw that before when I walked through some code. You declare when something happens, this is when I wanna run. So instead of directly invoking jobs, this allows you to decouple and also build things independently and do things like fan out, as Brian alluded to.

So the ingest system basically understands what your function, your workflows, want to do when, and it basically routes them through back to to, to your server. So at the highest level, that's how we're stitching together the hooks with events ingest and then back to the back to the system to to call your code.

Speaker 0: Beautiful. Beautiful. I I love the love the diagram, and it like, this was a a big reason why we shifted to this format to make it feel a little more interactive, a little more fun. So all that said. Right?

Now to the main event of, like, hey. We we've got this set up. These two are talking together. How are we going to build this advanced workflow? What are we gonna build?

So we're gonna show you guys content translations that are automatic and, of course, durable, and I will say totally unbreakable because I know I've written a lot of the code, but but very, very durable, and, like, an incredible workflow for, anybody on your content team that needs to handle translations. So if you've worked with content translations and other systems, this GIF probably holds true. It is so much fun, probably involving, like, a lot of spreadsheets, a lot of working back and forth with either different contractors or different team members. Directus makes that whole process a lot easier. We've got a beautiful interface for it, and, I think Matt on our team is gonna kill me for saying beautiful intuitive interface.

But let's dive into why this why this sucks. Right? We're gonna do this manually as kind of the first step that usually involves those spreadsheets, then we can graduate to APIs. The one we're using today is DeepL, but, you know, Chat GPT and some of the other LLMs have gotten incredibly good at translations. Kind of a toss-up there between speed, DeepL, I've found is is really quick, and, it seems to be highly accurate.

But as you get into that, especially when you're translating a lot of content, you run into, hey. This is gonna take a while. And as Dan alluded to, I've already ran into this, like, 35 times through the, building of this thing is, oops, we hit the rate limit of the APIs or one of those API calls fails. What happens? Right?

Then we lose not only our data, so, you know, we just hope and pray that there's some logs that we can go back and get some of that data, from their side or, you know, just evaporates. So, this is what we seek to fix inside this workflow. And before I show you Directus and Ingest, I want to take a look at our our data model at a high level. Now, this is a simplified example, and I guarantee you when you go to production, you've got a lot of content that you're gonna translate that it probably has a more complex structure than this. But, I do wanna give you this, like, again, we're gonna give you the whole code base.

You could take this and run with it, and so this starting point will make it easy for you to adapt to those, those more complex models. At the high level, we've got a post. The post has a title, a slug, and some content. Now, there's also a languages collection inside the Directus instance. Whenever you use our translations interface, we will create that for you if you don't have it already.

The format is pretty standard. We have a code like in US, the name English direction, and then one of the additions I've added here is just a DeepL translation code because their API, doesn't necessarily follow the standard ISO codes. And then for each, post that we're gonna translate we have a relationship to a collection or a SQL table called post translations. So within that we have a pointer back to our language and then we have our title, slug, and content. So at a high level that's the data model we're working with.

Let's kind of pull this up and and see what this actually looks like. Alright. Dan's gonna laugh at me here. I'm sure to mess up Arc here. Still haven't mastered the side by side in Arc.

So over here on the left, I've got our, ingest dev server up and running. If I go in here, you could see I I don't have any runs yet, But over here on the right is our special themed version of Directus just for the ingest webinar. I really dig the the black and green vibes that you guys have on the website, Dan, by the way.

Speaker 1: Same. Okay.

Speaker 0: Nice. Alright. So, if we take a look at this, right, this is a beautiful gosh. I gotta stop saying beautiful, Matt. Intuitive interface for all of our content editors.

We could see the default language up here at the top. So we're writing this in English. We're gonna pass that to the DeepL API. When we send this translation over, you know, I could set this and and write in whatever default language that I want, and we'll handle the translations. Directus gives you this beautiful side by side view for your translations, and we can see that all of these different languages in our system are fully translated.

So, the flow that we're gonna set up, and I'll I'll walk through this in a minute, is we're gonna take any post anytime a a change happens or we create a new post, we're gonna fetch all the languages that we have that we wanna translate content for, and we're gonna go and actually translate that. Beautiful. Alright. So let's take a look at this invincible translation workflow. And usually I do this kinda in code, at the top of my document.

I break this down. Since we did this format, I wanted to make it a little more visual, and I'll still walk you through the actual code for this. So at the start of this, what will happen and what I left off of this diagram is the user event. Right? User creates a new post, and then that kicks off the process here.

So we're gonna send the post ID, we call it a key, to this workflow. And the first step is gonna be normalizing these event keys because the Directus, event emitter, like if we do post dot update we get an array of keys versus like post dot create we just get a string for the key. So we're gonna normalize those keys. We'll check and see if relevant fields have changed. So if nothing has changed on an update, doesn't make any sense to actually build these translations again.

If something has changed, we're going to retrieve all the translations for that existing post, and that's an important piece. Again, you know, we don't wanna translate content if we don't have to, if it hasn't changed. This is gonna be using, Directus item service, so we'll dive into the code in just a moment. We have, we'll get all of our available languages via the direct Us item service again. We'll use both of those to build a list of translation items.

So what do we actually need to translate? We'll fire those off using, the ingest step functions to the DeepL API and we're gonna do that in parallel and we'll also loop over those so we get that nice tracking and observability. We'll log any errors and then when we get that back we're going to upsert into the database and potentially notify the user. So that's the flow at a high level. Let's take a look at the actual code.

And, Dan, if I gloss over something or I miss something on the ingest side, definitely call me out for it.

Speaker 1: Will do.

Speaker 0: Perfect. Alright. So, again, you can see here, this is, these are the two hooks that we're using to actually manage this workflow. So whenever an item gets updated, we send ingest, hey. We updated this, or we say, hey.

We created this. And then our function looks like this. And, this is not the shortest function I've ever written, but definitely not the longest. I'll say that. So, again, an outline of the translate workflow, what I found is super helpful for me is just to quickly outline the logic at the top of each one of these.

And if we go through, we could see that we're importing that ingest client. We've got the DeepL API, just their node client that we're gonna use, and we've got some types that we're pulling in to, make the TypeScript compiler TypeScript gods happy. So, we've got some translatable fields here. Again, it just defining some constants. These are the only fields that we wanna translate.

You could easily set this up to be dynamic, and just defining some of the the DeepL params that we're gonna use in our API call. So when we get into the meat of the ingest function here, you can see we've got an ID. We've got our name just describing what this is. And then, you can trigger an ingest function, on any number of events. So the standard syntax here is an object with an event property, but if you want to have multiple triggers for this function, you could just pass an array of objects.

Great. So onto our handler function, we're getting the event and the step, which is standard syntax and Jest is giving us that. And then we're also pulling out this direct us context that we added through that middleware. So through that, we're gonna get our services, we're gonna get our schema, our EMV variables, which is gonna give us the DeepL API key. So the first step in this function is normalizing those keys.

You could see all of that code here. What's notably missing is the step functions that we saw earlier. And originally, I had these wrapped, but, a a nice little tip that Dan gave me is if this, the code that you're running doesn't actually mutate external state or depend on external state, you don't necessarily have to wrap it in a in one of the ingest step functions. Next up, we'll get our payload from the event. We'll check that to see if we have any translatable content included.

And if we don't have any translated translatable content, then we can just return early. Right? Next, we'll get our translator. So this is the DeepL client. We'll get our schema from Directus, and we're gonna init these item services.

So this is how we talk to the database, on the Directus side of it. There's just a little helper function down at the bottom of this file that that makes that a little less verbose, and then we run into our first step function. So here, we're going to get the current post, and this is just a, a service call by the post service. So we're gonna read by the query. We're gonna look for the post with the ID that we were passed in the event and return not just the root level fields, but also the translations that are attached to that post.

So if we look at that in, like, the Directus UI to give you an example, we're not only gonna fetch this information, we're gonna fetch all the individual translations. And that is one of my favorite features of Directus is, being able to fetch the data that I need in a single API call. So I could go deeper into this if I wanted to, you know, three, four, five different levels. Eventually, you'll reach a max where you you don't wanna go, but, depending on the data that you've got, using these asterisks as a wildcard is incredibly helpful for local development. Now, going further, again, we're just going to have a step function that, will cancel this whole thing.

If we can't get the languages from Directus, we don't know what to translate. And this was a a last minute addition. And, Dan, could you talk about, like, the retry logic just a little bit?

Speaker 1: Yeah. As Ingest handles, errors automatically and does retries, some errors you might anticipate and say, this is a non retryable setup. So if I'm missing the API key, might as well not retry it because it's just not gonna work. So in that sense, you, ingest allows, includes a custom function, basically, which allows ingest to say, you know what? Let's stop here and not retry anything else.

So that's what Brian is using here where this is a nonrecoverable error kinda situation. So but, typically, you can throw errors to customers and whatnot, and, those will all be retried automatically. You can even catch them and handle them however you want as well.

Speaker 0: Beautiful. Beautiful. Thank you, Diane. Alright. Let me try to find where we were at.

Okay. So we've got our languages, then we move on to the next step, which is actually building our translation list. Excuse me, guys. Dealing with six not six kids, but sick kids here at the house. Always a struggle.

So here in this step function, we're basically just building up a list of the translations that we want. Right? So we're looping through all of those, posts that we've got, making sure, you know, we've got the fields that we wanna translate, and then we're basically building an array for those things that we'll we'll pass to the next step as we scroll down, which will be actually translating all these items in parallel. So, here, we're using promise dot all to fire these all at once. And, again, I think hey.

Dan, you mentioned this, like, it like, defining a unique ID for these was not not strictly necessary, but, maybe talk about that for a minute if you don't mind.

Speaker 1: Yeah. Yeah. When you're executing in a loop, it's or in something like here where you're paralyzing with promise dot all. You don't need to. Ingest basically takes this and under like, understands, like, internally.

You could check out though the SDK as well if you're curious about how it works. Basically, it takes the step ID and and and appends some sort of, iterator and creates a hash. So, automatically, it, make sure that if two steps have the same ID, it is it will, like, not they won't overwrite each other conflict. But for the sake of debugging, which I think is a great idea of what Brian's done here is you can dynamically set these keys, in your in your loop, which makes it easier for what Brian will show in just a minute, in the UI. Yeah.

Great.

Speaker 0: Alright. So we go through we send all these to DeepL. We get all of those things back, and then, the final two steps here, we're basically, again, using promises to do all of these upserts. So, when you're using these services in, like, inside the actual directest, like, API endpoints or hooks, there is this upsert, which, if anybody on the Directus team is listening, would love to have the upsert on the SDK. Just put in a nod for that one.

But, this makes it super easy to, upstart content. Super simple. Like, hey. If this ID doesn't exist, we're gonna create the translation here. And then last but not least, we're we want to notify the user, hey.

Your translations are done. So you can go check them out, because this is running in the background. Awesome. So that's the flow. Let's take a look at the UI.

How does this work? Let's, what do we what are we gonna translate? Dan, do you have any thoughts?

Speaker 1: I don't know. You hit someone good yesterday. It was just, like, hello from

Speaker 0: Hello from Dan and Bryant. Alright. So we're going to throw this in. What do we have? This is an amazing webinar.

This is not me. This is somebody in the chat. We're not saying we're amazing. But alright. Here we go.

So now what will happen as soon as I save this, over on the left, we should see the event being fired to ingest, unless I have done something totally wrong. And and just to prove that I'm not pulling any switcheroo on you guys, we don't see any translations there. So immediately after I've created that new post, now we could see, the ingest dev server and I get all this observability, all these steps that ran within the actual flow. So we get all of those steps broken down and you can see here we've got the individual steps within that loop that we use those specific IDs for. And as I go through here, you could see the output for each one of these.

So, I don't speak Russian. Not sure if you do, Dan, or not. No. So there we go. So we could see here's the actual content that's that's getting translated.

There's the slug. There's the title, etcetera, in all the different languages that we have set up. And this it's like seeing this together was, like, when it really hooked for me of, like, okay. Great. All this is running in the background.

I get all the observability. So, like, when something inevitably screws up, which for me is often, if you catch any of the hundred apps hundred hours episodes. But and, like, having all this at your fingertips is incredibly powerful as a developer. And being a a developer that has all this and is able to build a flow like this that will do all the translations for your team automatically, turns you into a hero, %. So that is the flow.

Dan, any anything to add before we kinda jump into queue?

Speaker 1: Yeah. And what's really nice is I think just, you set it up earlier with Docker Compose, but with everything running on your your machine, you can work and iterate quickly on this flow and not have to worry about, like, conflicting with, you know, bumping into shared resources. Like, if you're using something like SQS or something like that on Amazon, you need to provision those things. It becomes a little bit of a nightmare. And what also is nice here is, like, you know, Brian's code works perfectly as we see.

It's all green. But if there was an error, you'll be able to see that span go red. And what's nice about the dev server flow is that it saves the input of your function. So if Brian were to go back to his code base, fix that bug, save it, the dev server would basically you know, the his direct to server would would refresh, reload, and you could click rerun in this dev server, and it would just rerun the function again. So if he, you know, if you hit rerun live, it should just it should just work.

And so this gives you, like, this fast feedback loop. So you're, like, in this kind of hot reload situation where I'm working on my I'm tweaking I'm tweaking. So instead of you having to, like, go to the right, manually click a bunch of buttons in the in the Directus UI, you can have a fast feedback loop. So, like, do it once in there, keep going, you know, tweak the output, look at things, tweak maybe prompts or different things that you might be using, to to create this. So at least, like, this allows you to kind of, hopefully move a lot faster when you're building these these things that can be complex.

Speaker 0: But, yeah, I can't stress that enough. Like, the the speed at which you could iterate with the dev server and, direct us being able to, you know, go in and quickly model a feature and idea, and then also being able to, like, prepare that for scale using Ingest is, again, a great pair. Just, works really well together. Alright, guys. So if we move to our amazing slide deck, it is now time for Q and AO.

If you guys have any questions in the chat that we wanna take a look at, Dan, do you spot any that we need to

Speaker 1: I did see a couple questions I could talk to. The first thing I could do was a couple questions on self hosting for Jess side because we we know we can, self host direct us. Right? And I'm sure a lot of people do that as well.

Speaker 0: Yeah. Self hosting is incredibly popular for Directus.

Speaker 1: So I'm sure I'm sure there's a lot of people in the audience that are very curious about self hosting ingest as well. And you can self host ingest. The code that, Brian showed that runs in the DevServer, that binary is the exact same binary that you could self host. You can also you can run it in a very lightweight version or you can offload. Ingest has queuing and state history involved like that it that is backed by.

So when it's running locally, it just runs in memory because it's low volume and it's very simple. But when you self host it and you deploy it into your own cloud or wherever you want, you can hook it up to an existing, instance of Postgres or you can also, plug it out and and connect it to a dedicated Redis, maybe running in another container or something like that. It can handle a little bit more scale and handle, like, you know, restarts of your ingest system. So, you know, you can self host. There are certain things that aren't in self host yet, like, some observable observability and metrics.

A lot of those systems were built in ingest cloud. We're gonna be, you know, kind of bringing some of those things down to, to open source as well. But now, you know, all the key features, all the throttling, flow control, defining functions, are all are all there. So you can run that wherever you want and and and self host.

Speaker 0: Amazing. Got it. So I on the directed side of it, of course, like, you could self host, we've got a BSL license, which basically, a free to run for anybody under under $5,000,000 in total finances or revenue. So So if you have questions on the license, definitely reach out to our team about that. You can do that through the website.

What other questions do we have? I think there was one that I saw that, was super helpful. I like, comparing I like, when I first came into Ingest, I'd heard of Temporal, worked with it a bit. I can't find this one in the chat now, but, like, how do you guys stack up against temporal, Dan?

Speaker 1: Yeah. Yeah. We that's a I think it's a great question. You know, especially with the term durable execution, temporal's, you know, in its essence, describes itself as a durable execution engine. So it is dead focused on a, what we believe is just like the durable execution of the function.

Actually, the logic. Right? Is when something fails, it does checkpoints and it and it retries. So in that sense, there there is similarities. Right?

But we consider that durable execution is just like a means to an end. Right? It is it is a feature. It is not the whole platform. So what ingest really, layers on is a couple things.

We have an event based approach as Brian showed with the hooks, so you can fan out and you can replay and do more things, have a little bit more flexibility with events. And if you're someone who likes events, like, I'm sure that that resonates. And if you haven't, give it a try. And all the flow control and advanced queuing is one of the things that is unique to ingest and, is is in this self self hosted open source version as well is all this reliable flow control. So when people are building these systems, often you don't just wanna execute a job and run it to completion.

You need to manage how fast it's processing, this job, you know, how many times per minute you might run something. Maybe I wanna delay, I wanna debounce something, I wanna rate limit this job, run things in batch processing. Maybe just dynamically say, when there's 20 posts that have been published, let's execute this batch instead of saying, you know, let's just push these 20 items in one big blob. So there's a a a lot of differences in that sense of, like, what we've built around. And I think one of the things that also is true with the ingest SDK is it doesn't mess with the runtime.

If you have used temporal, you in the TypeScript or JavaScript, you know, SDK, temporal does something where they kind of, like, wrap your logic and certain things like random doesn't work. So there's some gotchas that's like, I don't know what's going on with this runtime or it's going to wrap some of your things and you might not it might just not be native code, but we've fundamentally chosen to build ingest, say, anything that you're using works. It's very easy to look in the source and see where things are running. It's very it aims to be a very thin layer, so there's no weird kind of, like, gotchas and things that I need to know. And, generally, like, you know, ingest is very dynamic and defining steps and everything is very fluid.

So there aren't, there's a lot of friction to, like, the rigidity that you might find with other solutions. So I those are just a few things. There's many more, that you could check out, like, on our site and whatnot, but or ping me afterwards if you if you if you're curious.

Speaker 0: Yeah. And I'm not sure that you could say this, but, like, the the syntax that that I found, like, writing the same sort of thing in in jest is dramatically, easier, and it just jives with the way that my brain works versus, like, some of the verbosity and, like, just how temporal structures things. So, do we have any other questions? I I guess that's gonna sting a bit if we have temporal onto one of the partner webinars as well. But

Speaker 1: I, what is called? I'll mention one someone asked a question about, retries. What happens when something hits hits max retries? The function will be declared as failed. And what, ingest also allows you to do is basically say, you know, if there's a complete outage, say, DeepL's down for twenty four hours or or an hour, all your functions are failing and all the retries are exhausted, you can use ingest to say, select, you know, between these two time stamps, anything that failed, replay them so you can do bulk retrace, retries, we call replays of, of those functions.

So you can recover from systems because we persist all the inputs, you know, if that's helpful.

Speaker 0: Definitely. And then, I think probably the last question before we wrap up is, can you run webhooks in the ingest server, the dev server?

Speaker 1: You, you can. You'll have to configure your webhooks in in in in I guess it depends on, like, how you're how you're running things. So, Ingest Cloud, it has webhooks and and transforms. And locally, you just need to write a little logic to just, like, wrap that transform to simulate what, what is happening. We'll be bringing in in the future a synchronization, some thing with Cloud to make that a little bit more easy and bring them into the DevServer.

But fundamentally, the webhooks are just the same API endpoint that ingest dot send is using, that Brian showed in the in those, in those direct hooks. So it's just sending JSON payloads, and that's what, what webhooks primarily are. So, it's, it's pretty easy to to utilize it and build around. There's also a few different docs on our website about that if you're curious about webhooks.

Speaker 0: Yeah. Awesome. I don't I don't see any other questions from the team. We're a little bit over, Dan, but I I I man, I appreciate you coming on. Like, before we kinda get into the awkward outro, I do want to just reiterate, like, for anybody still here, if you've registered for this, we'll hit you with an email with all the links to the repo, this amazing slide deck that we put together.

So don't worry about that. That'll be coming in the next, couple hours, so just be patient. But wrapping this thing up, we we hope this was helpful for you guys. Please send us your feedback. We really enjoy doing these webinars, showcasing other tools and, things that that help you build faster.

And, Dan, thank you for joining. Really appreciate the, you know, the collaborative effort on this thing. Learned a ton about it. And, for everybody else who like, next steps for you guys, what what does that look like? You know, you wanna sign off a little bit?

Speaker 1: Yeah. Yeah. Thanks for thanks for having me, and, thanks for doing all the leg work. You wrote all the code. You built all of it.

So it's, it was pretty awesome to see this, and I think it's a great example. I think if you really wanna figure it out and you want some more, there'll be the follow-up email. Brian also has a really great tutorial that he put together, which has a lot of detail. It's pretty incredible, but definitely check that out. I'm sure that'll be in the follow-up email.

And, then also, I'd say, like, if you just wanna tinker a little bit, go with ingest, go check out, one of the quick starts on our docs and just go tinker a little bit with the dev server, build some different things with some dummy code, and then kind of, that's easy way to just kinda get started, and then you can kind of dive in for the in-depth stuff with Directus using, using Brian's tutorial that he wrote. So thanks for everybody joining. And if you ever have any questions, we, we do have a Discord community. You can find the link on our site, or you can always reach out to us, contact us anytime with ingest dot com. That's 2, by the way.

Just always remember that one.

Speaker 0: Two n's. Yeah. Perfect. Alright, Dan. Thank you.

Thanks for the audience. That's a wrap. We'll see you all.

Speaker 1: Thank you.