How We Worklet: Data Extraction with Third-Party Storage

One of the most important and first steps in troubleshooting is gathering information. Typically for IT this means collecting system, service, or application logs.

Asking someone to find and upload these files can be confusing, take a lot of time, and in some cases seem outright impossible due to file or folder permissions based on your organization’s security policies.

The Data Extraction with Third-Party Storage Worklet was built to reduce time spent on information gathering, give power to the IT helpdesk to automatically gather information on devices, and eliminate frustrating back-and-forth communications.

Video Transcript:

Hi, thank you for joining me here today. My name is Colby Hall and I'm the System Administrator here at Automox.

Today we're going to be going through the Device log extraction with 3rd-party storage. Let's go ahead and walk through what it's like for an end user experience.

Here you see an end user with an empty desktop. We're gonna go through on the back end and run the device log Worklet. We're gonna first run it on the Automox console which is going to trigger everything. Following up, you'll see that on the end user's device the files will go through and get dumped onto the desktop pretty quickly.

We'll have two sets of files, zip files as well as JSON files with our zip files. We'll have one for each of the locations where we want to pull information from here. We have the Automox agent log files, system log files, as well as Netskope log files.

We'll also have the hosting identifier of the device for pulling the information from. We'll also have JSON files which are formatted to include Metadata information, which we can send up to Rapid7 so we can rename the files as well as have other info such as the encryption key for it.

Next, we're going to go through and see what it's like for the IT admin on the back end. Looking at google tribe. You can see here the logs went ahead and set up fairly quickly.

We still have the hosting and the type of file that we took, which is a system log file, Netskope log file, and Automox agent file. When we look at the information that we collected, we can go through and find the one that has the highest digit count as the most recent file.

We can also find the first pulled file or the most historic by looking at the lowest value for that first integer. It is important to note that this is on a shared drive. Here by default using the Rapid7 Google drive connector and just a limitation with the google drive API. You are only able to specifically target folder IDs for an individual user, not a shared drive folder ID. Nor to bypass this. You can actually set up a google apps script to get around this problem.

If you right-click any section in your google drive instance, you can actually go through this list of actions to the more to pull up another drop-down then google to google apps script. This is actually a scripting service that Google hosts to interact and automate certain actions within their applications such as google drive. When he clicked into that you could see we actually have one that we went ahead and created. This goes through and it has a variable called files which looks for and finds any item with a specific folder ID. Which is our origin folder ID under an individual user.

Next, we have a wild loop which will go through. If there are any files in that specific holder ID. We'll go through and then push them or copy them to a destination folder ID. Which can be in a shared folder instance and then remove the original files from starting folder ID utilizing this method we can drop files into a shared folder instance which other IT Members can also view next we'll go through and see how it's actually built out in the Automox console. As with all work in our Automox console, it's always broken down into an evaluation code and our mediation code for the evaluation code.

We're always setting our environment variable as bash. Next, we'll do an evaluation to see if we have a certain file on the system. You don't have to do this. You could just say exit one and have everything always go through and remediate which means there'll be no check but for us specifically we want to make sure that the device at least has the NetsKope application installed.

If it does, then I'll exit one and continue on with the remediation code. If not it will exit zero and not continue moving along to our mediation code. We do have some variables that we set that are pretty common in scripts.

We have the logged-in user which gets the person that's currently using the device as well as the date. However, remember that? This is not the day, month year format. We're actually just taking the second format. We also have a couple of variables for the final folder path or file path destination.

We also have the hostname variable which grabs the hostname of whatever device we're grabbing us from. Since we need this to run on any device that we want to rent it on.

Next. We'll go through and actually use the zip command to compress these files into their destination which were again we're using those variables for and here we are grabbing the system log file as well as the Automox agent log file for other applications. You can sometimes use a command included for this instance we use the NSD command and output it to the folder path variable nor to drop it off as the Netskope blog.

Next, let's go ahead and encrypt those files. So for each one of those file path variables, we'll go ahead and echo those file path names. Give us the files with their specific location and then we will Support them over to the base64 command which base 64 encrypts these files to be sent up securely. Otherwise, we wouldn't have them encrypted and anyone could intercept our transmissions of our data which is not a great thing.

Next, go ahead and format those files and create new JSON files to have identifiers for the stuff that we're sending up using the cat command. We have key-value pairs which are key identifiers that will include. So we have the file name which includes the actual name of the file that we wanted to have when it gets sent up to Google Drive.

So here we have that date variable the hosting variable as well as the full Netskope dot zip file name. We also have the base 64 data or where the encryption key is For those encrypted files. That way we can decrypt them on the Rapid7 end last but not least. We'll go ahead and use the curl command and then send those up with a couple of more identifiers with a content type identify as well as a character set which is UFF eight identifier. The file that we're gonna be including to identify these key-value pairs that we set up previously. And then we'll also go ahead and include the website URL which is the webhook API trigger included in our Rapid7 workflow For our safety. We're not including ours here as well as when you set it up via Rapid7. You will have your own unique API trigger webhook URL.

And let's go ahead and go into Rapid7 to see what that looks like. So here you can go ahead and see our full workflow that we run through via Rapid7 which is the next step in the process. Go through hit a webhook, API URL. Which is included when you set this up initially.

And then we're gonna go ahead and designate some starting variables. We have the foul name variable which is a string data type As well as a base 64 data variable which is again another string data type.

With those selected. We're going to go ahead and choose another action for our workflow. We're gonna be selecting the compression action by Rapid7. We're going to go ahead and extract the archive nor to un compress the information, you'll create a test orchestrator, that's just the automation that will run through the actions for you on your behalf, and then we'll go ahead and get some input for this step.

Well actually including all the variables from the previous step in this one so we can keep things nice or meet ordinary as well as non-complex nor to include any variable from a previous step will simply open up a text box, click on the little blue plus sign, lower bottom right.

And then for this instance, we need the basic c. four string as well as the file name string. Once we have that step saved we have one more step.

Gonna go ahead and choose an action. We're gonna go all the way down till we find the google drive connector and we're going to actually create a file in a folder. If we were going to upload a file in a folder it would have to be a specific file format.

But since we don't want anything that we send up to be automatically put into an Excel or Google Sheets file or google Powerpoint slides, we'll go ahead and create a file in a folder instead. We already have a connector set up but if you're going to set up your own um you would need some information from your google drive or google cloud platform instance. Um, we did put a link in the blog to show you how to set this up and where to set this up.

However, for our purposes we already have it connected. So go ahead and continue. You will need to go ahead and include your folder ID, which you grab from your google drive instance as well as include the variables from the previous steps which is the base 64 data as well as the file name with this set. All we need to do is activate and then run the initial work lit through the Automox console that will run through all the steps that you saw previously sending the files up to the customer's desktop setting those through the Rapid7 Worklet, as well as pushing them eventually up to Google Drive for you as an IT admin to utilize and make complete use of in order to assist with your troubleshooting process.

And with that, that is all the information that we have for this blog for you today. So I thank you again for joining me again. I'm Colby Hall with Automox, the System Administrator. Have a great day.