[{"data":1,"prerenderedAt":67},["ShallowReactive",2],{"quick-connect-deepgram":3,"quick-connect-deepgram-next":52},{"id":4,"slug":5,"vimeo_id":6,"description":7,"tile":8,"length":9,"resources":10,"people":14,"episode_number":18,"published":19,"title":20,"video_transcript_html":21,"video_transcript_text":22,"content":23,"status":24,"episode_people":25,"recommendations":36,"season":37,"seo":23},"a230c9ef-8db4-4c00-a0cb-9524f7934eb0","deepgram","895925213","Automatically transcribe new audio files with Deepgram's Speech-to-Text API.","f894ea9e-da61-4529-b51a-b399acd1bdc8",11,[11],{"name":12,"url":13},"Deepgram Docs","https:\u002F\u002Fdevelopers.deepgram.com\u002Fdocs",[15],{"name":16,"url":17},"Kevin Lewis","https:\u002F\u002Fdirectus.io\u002Fteam\u002Fkevin-lewis",2,"2024-01-05","Transcribe Audio Files with Deepgram","\u003Cp>Speaker 0: Deepgram offers a speech to text API that allows us to send off audio files and receive transcriptions in return. Today in QuickConnect, we're going to connect to Deepgram. So whenever we upload a file to our director's project, if it's an audio file, it will automatically go off, generate a transcription, and then save that transcription to the file description. So let's get started. The first thing we'll need is a Deepgram API key.\u003C\u002Fp>\u003Cp>So head to the Deepgram console and create a new API key. You can give it any name you want and set the permissions. We only require the lowest level member permissions. We'll set the API key to never expire and hit create key. I'm gonna take note of this API key for later because I only get to see it right now one time.\u003C\u002Fp>\u003Cp>If I lose this key, I'll need to come and generate a new one here in the Deepgram console. So hit got it and head over to your directors project. Let's create a new flow in our directors project. I'll call this one transcribe new audio files. We're going to set this up to trigger whenever an event happens in our directors project.\u003C\u002Fp>\u003Cp>We're going to pick non blocking, which means that all of directors' built in functionality will not be stopped or halted or paused, but this logic will run-in parallel. Now in terms of scope, we're going to pick files dot upload. So this will trigger every time a file is uploaded, and we will hit save. So that is our trigger. Now the first thing we're going to do is actually make sure that we only continue this logic if the file is an audio file.\u003C\u002Fp>\u003Cp>Because right now, this flow will start on every single file upload regardless of file type and location. So let's create a new operation. I'll just call it check and add some conditional logic. This condition ensures that the file type contains the word audio. The file type could be something like audio \u002Fmp3audio.wav and so on.\u003C\u002Fp>\u003Cp>But they all start with audio. So we're gonna check that the file type contains the word audio. Now there are 2 paths here, the resolve path and the reject path. Now if this condition is not true, I e, the file is not an audio file, we will go down the reject path, and we're just gonna add nothing there, which means the whole flow just just stops. It completes at this point.\u003C\u002Fp>\u003Cp>Let's create a new operation on the resolve track. And in here, we're going to call it Deepgram. We're gonna go ahead and make this a webhook slash request URL, operation, and this provides some additional options. Now in the Deepgram documentation, they give you a URL to use when you want to go ahead and create transcriptions. It needs to be a post request to api.deepgram.com\u002Fvone\u002Flisten.\u003C\u002Fp>\u003Cp>And then you can add any number of query parameters to change what Deepgram is going to do and return with that audio file. So here we're saying, basically, make it human readable, add some add some formatting, and diarization will split out who's speaking. So it will tell us whether it's speaker 1 or speaker 2 and so on. Now Deepgram also requires a header in order to authenticate. So we're gonna go ahead and add a new header.\u003C\u002Fp>\u003Cp>We're going to call this one authorization, capital a, and the value will be the word token, space, and then your Deepgram API key, and hit save. Now the final thing we need to do is actually give it the file URL. So what we're actually gonna do is pause here for just a moment, and we're going to try and trigger this flow as it stands. This will fail because we're not providing the file, but we're gonna see what this does. So I'm gonna go ahead and add a new file to our file library.\u003C\u002Fp>\u003Cp>Here we go. So we'll upload that file. Fantastic we will head back to our flow and we'll see here that there's a little one icon here one run and we'll get to see. So we get a trigger. So this is the actual file being uploaded.\u003C\u002Fp>\u003Cp>Now there's an interesting thing here. We see inside of this trigger value, there is a key. This is the unique identifier for this file in the director system. Then we see here that this was the check, and then we did a Deepgram API call with our header. And the returned payload, as we expected, is incorrect because we didn't actually send the file.\u003C\u002Fp>\u003Cp>The reason I wanted to do this is because I wanted to show you that we need this ID trigger dot key. So now let's go in and edit this, this request over here. I'm I'm gonna just paste it in, and we're gonna explain it here. So every asset in Directus is accessible with the full URL of the Directus project \u002Fassets\u002Fthekey, the unique ID for that file. So this is now the URL for that file.\u003C\u002Fp>\u003Cp>If I go to this complete URL, of course, putting the real key in here, I would actually get that file. However, there is one extra thing to note here, which is that right now as it stands, this directors project does not make all of the files public. Right now, if we continue with this request and we run that again, we reupload a file, there'll be an authentication error. Directus won't be able to reach that file. Then we have a few options.\u003C\u002Fp>\u003Cp>The first is to go into the public role, go into directors files, and make that file or the directory which the file lives in public. That's one viable option. But there is another way to do this, which perhaps is a little more recommended. Let me go into my user account here and generate an API token. So I will save that and hit save.\u003C\u002Fp>\u003Cp>Going back to my flow here, I'm gonna once again edit this URL, and I'm going to append to the end of it access_token, and then my API key. This will authenticate the API request. So this now is a file that I can access. Let's trigger this flow again. Let's go over to our file library.\u003C\u002Fp>\u003Cp>I'm actually just gonna go ahead and re upload the file again. So it's a fresh file. So we'll have a duplicate here, but that will trigger the flow to rerun. Let's look at this second run of the flow. The returned payload comes back and there is our transcript.\u003C\u002Fp>\u003Cp>That's fantastic. So we get that back. So our Deepgram request was successful. There's actually quite a large payload that comes back here. Data, results, channels, which is an array of objects, alternatives, which is an array, and there is our transcript.\u003C\u002Fp>\u003Cp>Because we turned on smart formatting, there is also a ton of extra metadata that comes back. Like, it's huge. Every single word, every single word there. And we also get our paragraphs coming back with a transcript and so on and so forth. So you can go and explore this returned object.\u003C\u002Fp>\u003Cp>I, surprise, surprise, have already done that work for us. The final step is going to be to actually save this value back to the file description. So let's create a new operation here, and let's make this an update data, operation. In the collection field, we are going to update a director's files item. And we're going to put in here the ID of that initial item trigger dot key.\u003C\u002Fp>\u003Cp>So now we're gonna be updating the item that actually triggered this whole flow in the first place and in turn was transcribed. Now the final thing we're gonna do here is we are going to update the description. So let's go ahead and do that here together. The description, which is a built in field, is going to be equal to Deepgram and the reason we can say Deepgram is because that is the key of this step it's called Deepgram Deepgram dot data dot results dot channels. That was an array Dot alternatives.\u003C\u002Fp>\u003Cp>That was an array.paragraphs dot transcript. Oh, rolls right off the tongue that. And, of course, that might be a little bit of development trial and error, but I know that that is the location of the formatted transcript. Let's hit save. Let's hit save again.\u003C\u002Fp>\u003Cp>And now let's go and upload once again a new file. So that's uploading. Now in the background, that flow has already been triggered. Deepgram has or the condition has already been checked. Deepgram has already been called.\u003C\u002Fp>\u003Cp>And in theory, if we go in here now, we should see in the description did I pick the wrong file? Yes. I did. It was this one. The description with speaker diarization and formatting.\u003C\u002Fp>\u003Cp>So, automatically, now that this flow is set up and enabled, I can go hands off and know that every audio file that gets uploaded will be transcribed. Now at the time of recording, Deepgram do also accept video files as well, but you can expect that they'll take a little bit longer to return. And, typically, you should be trying to send us as little, you know, data as possible. So audio files was our condition, but you can widen that condition as well. So let's just sum up the way that that flow looks.\u003C\u002Fp>\u003Cp>Transcribe new audio files. This is triggered whenever a new file is uploaded. And just to be clear, that's every file. So what we do is we check that it's an audio file, then we go off to the Deepgram API authenticating with our API key. And then once we get the transcript back, we update this file with the transcript.\u003C\u002Fp>\u003Cp>Now there are some extra things just to consider here. Right now, this will happen on every audio file. You can add further conditions to perhaps specify down to a folder or only allow specific users to do this, or you can make it a manual step. You can set up a flow with a button on an item page and go ahead then and only generate a transcript on demand. I hope you found this interesting, and I'll see you in the next episode.\u003C\u002Fp>","Deepgram offers a speech to text API that allows us to send off audio files and receive transcriptions in return. Today in QuickConnect, we're going to connect to Deepgram. So whenever we upload a file to our director's project, if it's an audio file, it will automatically go off, generate a transcription, and then save that transcription to the file description. So let's get started. The first thing we'll need is a Deepgram API key. So head to the Deepgram console and create a new API key. You can give it any name you want and set the permissions. We only require the lowest level member permissions. We'll set the API key to never expire and hit create key. I'm gonna take note of this API key for later because I only get to see it right now one time. If I lose this key, I'll need to come and generate a new one here in the Deepgram console. So hit got it and head over to your directors project. Let's create a new flow in our directors project. I'll call this one transcribe new audio files. We're going to set this up to trigger whenever an event happens in our directors project. We're going to pick non blocking, which means that all of directors' built in functionality will not be stopped or halted or paused, but this logic will run-in parallel. Now in terms of scope, we're going to pick files dot upload. So this will trigger every time a file is uploaded, and we will hit save. So that is our trigger. Now the first thing we're going to do is actually make sure that we only continue this logic if the file is an audio file. Because right now, this flow will start on every single file upload regardless of file type and location. So let's create a new operation. I'll just call it check and add some conditional logic. This condition ensures that the file type contains the word audio. The file type could be something like audio \u002Fmp3audio.wav and so on. But they all start with audio. So we're gonna check that the file type contains the word audio. Now there are 2 paths here, the resolve path and the reject path. Now if this condition is not true, I e, the file is not an audio file, we will go down the reject path, and we're just gonna add nothing there, which means the whole flow just just stops. It completes at this point. Let's create a new operation on the resolve track. And in here, we're going to call it Deepgram. We're gonna go ahead and make this a webhook slash request URL, operation, and this provides some additional options. Now in the Deepgram documentation, they give you a URL to use when you want to go ahead and create transcriptions. It needs to be a post request to api.deepgram.com\u002Fvone\u002Flisten. And then you can add any number of query parameters to change what Deepgram is going to do and return with that audio file. So here we're saying, basically, make it human readable, add some add some formatting, and diarization will split out who's speaking. So it will tell us whether it's speaker 1 or speaker 2 and so on. Now Deepgram also requires a header in order to authenticate. So we're gonna go ahead and add a new header. We're going to call this one authorization, capital a, and the value will be the word token, space, and then your Deepgram API key, and hit save. Now the final thing we need to do is actually give it the file URL. So what we're actually gonna do is pause here for just a moment, and we're going to try and trigger this flow as it stands. This will fail because we're not providing the file, but we're gonna see what this does. So I'm gonna go ahead and add a new file to our file library. Here we go. So we'll upload that file. Fantastic we will head back to our flow and we'll see here that there's a little one icon here one run and we'll get to see. So we get a trigger. So this is the actual file being uploaded. Now there's an interesting thing here. We see inside of this trigger value, there is a key. This is the unique identifier for this file in the director system. Then we see here that this was the check, and then we did a Deepgram API call with our header. And the returned payload, as we expected, is incorrect because we didn't actually send the file. The reason I wanted to do this is because I wanted to show you that we need this ID trigger dot key. So now let's go in and edit this, this request over here. I'm I'm gonna just paste it in, and we're gonna explain it here. So every asset in Directus is accessible with the full URL of the Directus project \u002Fassets\u002Fthekey, the unique ID for that file. So this is now the URL for that file. If I go to this complete URL, of course, putting the real key in here, I would actually get that file. However, there is one extra thing to note here, which is that right now as it stands, this directors project does not make all of the files public. Right now, if we continue with this request and we run that again, we reupload a file, there'll be an authentication error. Directus won't be able to reach that file. Then we have a few options. The first is to go into the public role, go into directors files, and make that file or the directory which the file lives in public. That's one viable option. But there is another way to do this, which perhaps is a little more recommended. Let me go into my user account here and generate an API token. So I will save that and hit save. Going back to my flow here, I'm gonna once again edit this URL, and I'm going to append to the end of it access_token, and then my API key. This will authenticate the API request. So this now is a file that I can access. Let's trigger this flow again. Let's go over to our file library. I'm actually just gonna go ahead and re upload the file again. So it's a fresh file. So we'll have a duplicate here, but that will trigger the flow to rerun. Let's look at this second run of the flow. The returned payload comes back and there is our transcript. That's fantastic. So we get that back. So our Deepgram request was successful. There's actually quite a large payload that comes back here. Data, results, channels, which is an array of objects, alternatives, which is an array, and there is our transcript. Because we turned on smart formatting, there is also a ton of extra metadata that comes back. Like, it's huge. Every single word, every single word there. And we also get our paragraphs coming back with a transcript and so on and so forth. So you can go and explore this returned object. I, surprise, surprise, have already done that work for us. The final step is going to be to actually save this value back to the file description. So let's create a new operation here, and let's make this an update data, operation. In the collection field, we are going to update a director's files item. And we're going to put in here the ID of that initial item trigger dot key. So now we're gonna be updating the item that actually triggered this whole flow in the first place and in turn was transcribed. Now the final thing we're gonna do here is we are going to update the description. So let's go ahead and do that here together. The description, which is a built in field, is going to be equal to Deepgram and the reason we can say Deepgram is because that is the key of this step it's called Deepgram Deepgram dot data dot results dot channels. That was an array Dot alternatives. That was an array.paragraphs dot transcript. Oh, rolls right off the tongue that. And, of course, that might be a little bit of development trial and error, but I know that that is the location of the formatted transcript. Let's hit save. Let's hit save again. And now let's go and upload once again a new file. So that's uploading. Now in the background, that flow has already been triggered. Deepgram has or the condition has already been checked. Deepgram has already been called. And in theory, if we go in here now, we should see in the description did I pick the wrong file? Yes. I did. It was this one. The description with speaker diarization and formatting. So, automatically, now that this flow is set up and enabled, I can go hands off and know that every audio file that gets uploaded will be transcribed. Now at the time of recording, Deepgram do also accept video files as well, but you can expect that they'll take a little bit longer to return. And, typically, you should be trying to send us as little, you know, data as possible. So audio files was our condition, but you can widen that condition as well. So let's just sum up the way that that flow looks. Transcribe new audio files. This is triggered whenever a new file is uploaded. And just to be clear, that's every file. So what we do is we check that it's an audio file, then we go off to the Deepgram API authenticating with our API key. And then once we get the transcript back, we update this file with the transcript. Now there are some extra things just to consider here. Right now, this will happen on every audio file. You can add further conditions to perhaps specify down to a folder or only allow specific users to do this, or you can make it a manual step. You can set up a flow with a button on an item page and go ahead then and only generate a transcript on demand. I hope you found this interesting, and I'll see you in the next episode.",null,"published",[26],{"people_id":27},{"id":28,"first_name":29,"last_name":30,"avatar":31,"bio":32,"links":33},"82b3f7e5-637b-4890-93b2-378b497d5dc6","Kevin","Lewis","a662f91b-1ee9-4277-8c9d-3ac1878e44ad","Director of Developer Experience at Directus",[34],{"url":17,"service":35},"website",[],{"id":38,"number":39,"year":40,"episodes":41,"show":49},"3b8b7d34-a0fb-4ea6-85ff-2b5bfbb8e0b6",1,"2023",[42,4,43,44,45,46,47,48],"502dcf7e-c23e-4dfd-b147-65f5abaea5c7","5f41dc16-29b7-485f-a6e1-081c3f1acc4f","81417d25-26d2-4f05-be37-7ced51a0594e","8f933ee9-4e4f-4e35-8c1f-e99ad0684bfa","71e081db-92f8-4978-b020-7d2460a46187","8e47020d-bd5a-43a7-bca9-54af4f5d465d","bfb8bc25-ef1b-4544-b50d-402008c638a1",{"title":50,"tile":51},"Quick Connect","1171b046-491e-4cfb-a68c-527b89c2c348",{"id":53,"slug":54,"season":55,"vimeo_id":56,"description":57,"tile":58,"length":59,"resources":23,"people":23,"episode_number":39,"published":60,"title":57,"video_transcript_html":61,"video_transcript_text":62,"content":23,"seo":63,"status":24,"episode_people":64,"recommendations":66},"979db4da-a870-4120-94ee-bd80789f411c","firecrawl","cf7a056d-fa10-4bc5-8cc3-c2b9ef59b684","1026203173","Integrating Firecrawl with Directus","d0a87153-8475-433f-aca0-dea9802caf03",10,"2024-10-16","\u003Cp>Speaker 0: Hello there. I'm really excited about this tutorial. So on Directus TV, we already have a show called Quick Connect, which shows you how to integrate third party services with Directus using Directus Automate and Flows. And in the spirit of that show, today, I'm gonna show you how to integrate FireCrawl with Directus. Now here they say that they turn websites into LLM ready data.\u003C\u002Fp>\u003Cp>And what that means in practice is you can feed it a URL, provide some options if you want, and it will go and take a look at that web page and return some structured data for you like so. This is their scrape endpoint, which will take a single web page and scrape some data from it. They also have a couple of other endpoints, crawl and map, but today, we're gonna use scrape. Now I've already logged into FireCrawl Cloud and generated an API key, which I'll copy for later. You can also self host FireCrawl, but for ease, I'm just gonna use their cloud product here.\u003C\u002Fp>\u003Cp>Now I have this directors project over here with a new empty collection called companies. In this collection, there are a few fields. URL, a name, a description, mission, and a boolean, a true force value, is it open source. And our goal will be to provide the URL and then have FireCrawl automatically populate the rest with flows. So let's go ahead and create a new flow.\u003C\u002Fp>\u003Cp>So this is our automation builder if you've not seen it before. I'll call this one get company data, I guess. And we are going to use a manual trigger, which will add a button to the side of collection and item pages. So we're going to say we'll run this on the company's collection. What else matters here?\u003C\u002Fp>\u003Cp>We're going to, not require selection, so the button always works. And we're going to require confirmation, which will pop up a modal. And in that modal, we will just add a URL. We'll make it a string input, and we'll make it full widths. And I think that's all we need to do here.\u003C\u002Fp>\u003Cp>So just to see what happens here, if I go back to the, to the company's collection, we now see this button here, this manual flow, trigger. I click that. It pulls up the box, and we'll put in a URL and hit run flow. So now if we go back to our flow, we should immediately see that there is one log. And in here, there is a body, and the URL is the value that we typed.\u003C\u002Fp>\u003Cp>Fantastic. Now we need to actually do something with it. So, let's go ahead and add a new operation here. And, honestly, fire crawl is pretty sick. You can just make one web request.\u003C\u002Fp>\u003Cp>Let's take a look at their docs. We're gonna use the LLM extract, endpoint here, and let's just take a look at the kind of construction of this API call. It's a post request to this URL. We're gonna pass in our, our API key here as an authorization header, and then they give us this kind of JSON payload here. Here, it's telling it to go ahead and extract specifically these four fields, the company mission, does it support SSO, is it open source, and is it in Y Combinator?\u003C\u002Fp>\u003Cp>And it's saying you must go get all four of these. So let's actually just turn this straight into a flow request, request URL, operation. So we're gonna do a post request to this URL, post request to this URL. I'm gonna go and copy my API key again here, and at the end of this I'll, I'll destroy the key. Authorization authorization, bearer API key, save, And then there's the request body.\u003C\u002Fp>\u003Cp>And, honestly, it contains a little more than we need, but this contains everything we need. So we'll just pop that in there. The only thing we wanna do, of course, is pass in the URL that we put in the box. So we'll replace this with trigger.body. URL.\u003C\u002Fp>\u003Cp>Fantastic. Let's save that and see what happens if we go over to content, press the button, and type in directors.io. We see that's running. That's running. That's a good sign.\u003C\u002Fp>\u003Cp>It means it's going off and making the request, waiting for the request. And then we see there is a second log. And we get some data back. There was a 200, so it was successful. Inside of data, there is a property called data, and then there is this value called extract.\u003C\u002Fp>\u003Cp>Extract contains all of those custom keys we asked for, company mission, supports SSO, is open source, and is in YC. And then always when you scrape, you get this metadata object, title, description, language, Open Graph data, source URL, and so on. So, really, all we wanna do here now is we wanna take this data and create a new company from it. So let's add a new, let's create a new operation on the resolve path of that web request. Let's call it create data.\u003C\u002Fp>\u003Cp>We're gonna create something in the company's collection. We'll give it full access, and then we just need to provide a payload. So let's go ahead and do that. We have an object here. We have a name.\u003C\u002Fp>\u003Cp>Oh, we have a name, and we're gonna pass in the value of the last operation dot data.data.metadata dot title. Then I, for one, am I'm just gonna copy this and edit it each time. So we have name, URL. So that's last .data.data.metadata. Source URL.\u003C\u002Fp>\u003Cp>We could, of course, just take it from the trigger body URL, but this is properly formatted. You'll notice I typed in directus dot I o, but when it came back in the payload, it came back with the with the, with the protocol HTTPS and so on. So we have name. We have URL. We have a description.\u003C\u002Fp>\u003Cp>Now this one is also from the metadata description. We have the mission. Now this was a custom piece of data we asked to be extracted. Company mission is what we called it. And finally, we have open, open underscore source.\u003C\u002Fp>\u003Cp>Last data, data extract and then is underscore open underscore source and then remove that trailing comma. So I believe that's the name of all of the fields. We'll figure it out in a moment when when it inevitably doesn't work. We'll hit the button again, directors.io, and hit run flow. Once again, that's going off to fire crawl using their endpoint and there we see there we see it straight here.\u003C\u002Fp>\u003Cp>URL name, description, mission, and the boolean is open source. Let's, let's try that once more. Let's go in here and say firefirecrawl.devrun flow. So let's see. And in theory, we should just give that a moment, and there it is.\u003C\u002Fp>\u003Cp>So now you can go ahead and grab more data. Now, of course, if we take a look at this endpoint here, you can provide custom properties, and it will try its best to get data out from that. They have a couple of other interesting things which I'll draw your attention to even if I don't think it works in this context. They have extracting without a schema. So this extract here was us creating a a schema.\u003C\u002Fp>\u003Cp>Right? You can give it just a text. You can give it a prompt. Extract the company mission from the page. But the thing I don't like about that is you're not explicitly saying what the name of the key is, so you don't necessarily know what it's gonna be at the end.\u003C\u002Fp>\u003Cp>I like creating a schema personally. They do something else that's kinda interesting. If I take a look at where is I think it's in their API API reference here inside of scrape. They have this interesting thing called actions. So you can get it to wait, to take a screenshot, to click, write text, press a key, and scroll.\u003C\u002Fp>\u003Cp>And the combination of clicking and writing text means you can get it to interact with your web page. You see it here, actions, wait two milliseconds. You could get it to, like, sign into things perhaps, perform search terms. I think it's super interesting. And then take screenshots, of course, and upload those to directors if you fancy.\u003C\u002Fp>\u003Cp>So there's a lot of flexibility in this. Having seen kinda how easy this API is, I think I'll go ahead and turn this into, an extension some point in the next few weeks, which we can release as part of Directus AI. But, yeah, that's how to integrate FireCrawl with Directus using Directus Automate. Hope you found this interesting, and by all means, if you have questions, just reach out.\u003C\u002Fp>","Hello there. I'm really excited about this tutorial. So on Directus TV, we already have a show called Quick Connect, which shows you how to integrate third party services with Directus using Directus Automate and Flows. And in the spirit of that show, today, I'm gonna show you how to integrate FireCrawl with Directus. Now here they say that they turn websites into LLM ready data. And what that means in practice is you can feed it a URL, provide some options if you want, and it will go and take a look at that web page and return some structured data for you like so. This is their scrape endpoint, which will take a single web page and scrape some data from it. They also have a couple of other endpoints, crawl and map, but today, we're gonna use scrape. Now I've already logged into FireCrawl Cloud and generated an API key, which I'll copy for later. You can also self host FireCrawl, but for ease, I'm just gonna use their cloud product here. Now I have this directors project over here with a new empty collection called companies. In this collection, there are a few fields. URL, a name, a description, mission, and a boolean, a true force value, is it open source. And our goal will be to provide the URL and then have FireCrawl automatically populate the rest with flows. So let's go ahead and create a new flow. So this is our automation builder if you've not seen it before. I'll call this one get company data, I guess. And we are going to use a manual trigger, which will add a button to the side of collection and item pages. So we're going to say we'll run this on the company's collection. What else matters here? We're going to, not require selection, so the button always works. And we're going to require confirmation, which will pop up a modal. And in that modal, we will just add a URL. We'll make it a string input, and we'll make it full widths. And I think that's all we need to do here. So just to see what happens here, if I go back to the, to the company's collection, we now see this button here, this manual flow, trigger. I click that. It pulls up the box, and we'll put in a URL and hit run flow. So now if we go back to our flow, we should immediately see that there is one log. And in here, there is a body, and the URL is the value that we typed. Fantastic. Now we need to actually do something with it. So, let's go ahead and add a new operation here. And, honestly, fire crawl is pretty sick. You can just make one web request. Let's take a look at their docs. We're gonna use the LLM extract, endpoint here, and let's just take a look at the kind of construction of this API call. It's a post request to this URL. We're gonna pass in our, our API key here as an authorization header, and then they give us this kind of JSON payload here. Here, it's telling it to go ahead and extract specifically these four fields, the company mission, does it support SSO, is it open source, and is it in Y Combinator? And it's saying you must go get all four of these. So let's actually just turn this straight into a flow request, request URL, operation. So we're gonna do a post request to this URL, post request to this URL. I'm gonna go and copy my API key again here, and at the end of this I'll, I'll destroy the key. Authorization authorization, bearer API key, save, And then there's the request body. And, honestly, it contains a little more than we need, but this contains everything we need. So we'll just pop that in there. The only thing we wanna do, of course, is pass in the URL that we put in the box. So we'll replace this with trigger.body. URL. Fantastic. Let's save that and see what happens if we go over to content, press the button, and type in directors.io. We see that's running. That's running. That's a good sign. It means it's going off and making the request, waiting for the request. And then we see there is a second log. And we get some data back. There was a 200, so it was successful. Inside of data, there is a property called data, and then there is this value called extract. Extract contains all of those custom keys we asked for, company mission, supports SSO, is open source, and is in YC. And then always when you scrape, you get this metadata object, title, description, language, Open Graph data, source URL, and so on. So, really, all we wanna do here now is we wanna take this data and create a new company from it. So let's add a new, let's create a new operation on the resolve path of that web request. Let's call it create data. We're gonna create something in the company's collection. We'll give it full access, and then we just need to provide a payload. So let's go ahead and do that. We have an object here. We have a name. Oh, we have a name, and we're gonna pass in the value of the last operation dot data.data.metadata dot title. Then I, for one, am I'm just gonna copy this and edit it each time. So we have name, URL. So that's last .data.data.metadata. Source URL. We could, of course, just take it from the trigger body URL, but this is properly formatted. You'll notice I typed in directus dot I o, but when it came back in the payload, it came back with the with the, with the protocol HTTPS and so on. So we have name. We have URL. We have a description. Now this one is also from the metadata description. We have the mission. Now this was a custom piece of data we asked to be extracted. Company mission is what we called it. And finally, we have open, open underscore source. Last data, data extract and then is underscore open underscore source and then remove that trailing comma. So I believe that's the name of all of the fields. We'll figure it out in a moment when when it inevitably doesn't work. We'll hit the button again, directors.io, and hit run flow. Once again, that's going off to fire crawl using their endpoint and there we see there we see it straight here. URL name, description, mission, and the boolean is open source. Let's, let's try that once more. Let's go in here and say firefirecrawl.devrun flow. So let's see. And in theory, we should just give that a moment, and there it is. So now you can go ahead and grab more data. Now, of course, if we take a look at this endpoint here, you can provide custom properties, and it will try its best to get data out from that. They have a couple of other interesting things which I'll draw your attention to even if I don't think it works in this context. They have extracting without a schema. So this extract here was us creating a a schema. Right? You can give it just a text. You can give it a prompt. Extract the company mission from the page. But the thing I don't like about that is you're not explicitly saying what the name of the key is, so you don't necessarily know what it's gonna be at the end. I like creating a schema personally. They do something else that's kinda interesting. If I take a look at where is I think it's in their API API reference here inside of scrape. They have this interesting thing called actions. So you can get it to wait, to take a screenshot, to click, write text, press a key, and scroll. And the combination of clicking and writing text means you can get it to interact with your web page. You see it here, actions, wait two milliseconds. You could get it to, like, sign into things perhaps, perform search terms. I think it's super interesting. And then take screenshots, of course, and upload those to directors if you fancy. So there's a lot of flexibility in this. Having seen kinda how easy this API is, I think I'll go ahead and turn this into, an extension some point in the next few weeks, which we can release as part of Directus AI. But, yeah, that's how to integrate FireCrawl with Directus using Directus Automate. Hope you found this interesting, and by all means, if you have questions, just reach out.","9c575c49-4fb1-4eed-9731-e1e1acb55da3",[65],"4830ae03-22cd-4399-82f2-5767bf5cf0cc",[],1781877743301]