Today you are going to learn how to do bulk or batch uploads of information from CSV files on the ClearBlade platform. The bulk or batch uploads may be raw data collections, they may be new users that you want onboard, or perhaps it is new devices coming from your manufacturer that you want to put into the field.

Concepts Demonstrated:

• Configure device data functions to bulk uploads using CSV files

Prerequisites:

• ClearBlade Sandbox Account (Create)

1. Like always, the IoT Package Manager is the place to start from to find reusable packages. This is where you will search for the term “bulk” and find the “bulk-upload-package”. Clicking through it will show the installation instructions.

2. Start by clicking copy on the URL of the public repo for this package.

On the platform page, you can see in the demo that I have logged in as a developer and have several IoT solutions already up and running. Since this is a demonstration rather than importing these tools in existing package I am just importing it as a new system entirely.

3. Install this as a new solution by clicking “install” then pasting the URL and clicking “Fetch!” then “Import”.

Now that system is imported you can see there is a new system called “bulk upload” here and the next thing you’ll do is review the setup instructions.

For this IPM package there are only 2 setup steps. The first is to create the user that will do the import activities.

4. Go to your code service “Batch_setup” by clicking “code” under “bulk upload”, then click “Batch_Setup”.

5. You will notice it has created a username and password for the new user being created. You can change the password and then click “Save and Test”.

A popup will say “Success” if you have done everything correctly.

With that run we can see in the users table that the uploader user is now created by clicking “Users” in the left side bar.

That’s it for setup, so now we will open our bulk upload portal.

6. You can now log into the portal with your uploader account.

You’ll notice in the demo I am immediately given the opportunity to log in again with a developer account. I did that because there are a few things I wanted to be able to do in this portal that users of a system are not allowed to no matter how many privileges they are granted. Specifically, a user cannot pull the list of all the collections, this is a developer activity. It would make sense to grant this authority to your users. I only added this case, because it’s nice to be able to view the list of collections in a drop down picker below.

Now that you are logged in as the uploader and the developer, you can get started doing some bulk imports. First into a data collection. In this portal there is a preloaded jQuery library called jQuery CSV created by Evan Plaice [https://github.com/evanplaice/jquery-csv]. It’s amazing how great portals are when it comes to pulling in other libraries.

7. With that library pre-imported for you, the next step is to import a CSV file. There are some sample CSV files in the original GitHub link [https://github.com/aallsbrook/bulk-upload]. For our this demo we will be using the file “collectionCSV.csv” from the GitHub page. Click Choose Files, select the file you want to import, and then click Open.

If you look at the table, you can see that it contains a first row with column names and then many rows of new data. This bulk import IPM assumes that the headers will match your import collection columns. Bulk import requires that we make at least some assumptions about the data, but in this case you own the code and you can change those assumptions as needed.

Now you need to select the target import. In this case you want to put the data directly into a collection. You can quickly look in the platform by clicking Data then ImportCollection to see the import demo collection that you want to use and can verify that the column names all match. Right now it is empty, which is what you want.

8. To run the upload, under “3. Select type of asset being imported” select “Collection” and underneath it select “ImportCollection”. Then, click Import. If your import is successful, underneath “4. Run the import” it will say “import successful, see item_id below” and show you the unique item ids it has generated.

If you go back into the platform and click Data then ImportCollection you can see that the data did upload.

9. You can go a step further and try this same technique for users. For the demo we will run a quick reset on the portal and start the use case over. In this case, select a different CSV. For this user case we will be using the file “usersCSV.csv” that can be found in the original GitHub link where we downloaded the collectionsCSV.csv file. Click Choose File, select your user CSV, hit Open. Under “3. Select type of asset being imported” choose “Users”.

Before we import there are a number of things you’ll want to note. This user CSV has a first row that matches the column headers of the user table. The other thing to notice is that this CSV doesn’t contain passwords. To protect the confidentiality, it makes sense to try and keep those from ever being read by administrator or codified.

You can verify the schema and state of the users table before you start by looking in the platform under Users. With everything set you can run the bulk user upload by clicking Import.

You’ll notice this process takes a bit longer than the collections. That’s because ClearBlade treats that password with incredible security. Specifically, the generated password isn’t just being created, its being hashed with unique SALTed so to speak many many times. Again, if you’re import is successful you’ll receive a successful response. In the demo you’ll notice the passwords were sent back. This is good enough for the IPM, but I highly recommend you alter this example and rather than send back passwords, do something like directly email or SMS text your new users with their passwords to completely obfuscate you, the developer, ever having visibility.

Check out IPM for a number of mailer or communication IPMs for pre-configured libraries.

To validate the user import, take a look at the users table in the Users tab on the platform and you can see the newly created accounts in the system.

Last use case is a bulk device upload. Again, in the demo I reset the portal. This use case is a slightly different in the sense that here we are creating device records that store information about devices. A unique aspect to devices is the idea of enablement.

10. The initial process is the same as before using a collection CSV file. Click Choose File, select your collection CSV, hit Open. Under “3. Select type of asset being imported” choose “Devices”.

Notice the enabled column in the table. An enabled device can log in and perform transactions against the system while a disabled device will still have its unique record and state, but it can never log in. This disabled status is nice if you have devices that are manufactured, but not yet deployed or if you have devices in the field you want to simply turn off. The enabled flag is something we can send in the CSV file.

The demo collection CSV file has some custom columns and as such we customized the device table to match. The other thing to note, is that instead of passwords, devices have active keys. To get an auth token as a device, a device name and the active key make up the credential. This example assumes that we will generate keys during the import process. However, they could be passed in the CSV file if that was desired.

Now, click Import. You’ll notice this process is more compute intensive as each key is run through the SALT mechanism. If successful, you’ll receive a successful response and a display of the generated active keys.

 

That’s it for doing CSV bulk uploads.

Thanks for joining us here on the ClearBlade Technical Blog series!