Ultimate Guide to Edge Computing & Enterprise IoT – ClearBlade Best Practices
January 14, 2017
Welcome to the ClearBlade IoT platform. This document contains a number of helpful hints for developers creating and building IoT solutions. As a start, please check out the docs page where many concepts and tasks are explored in greater depth.
Code
Basics
- Code is designed for high scale / high performance real-time execution. As you author a service — consider its behavior at scale being called 1000’s of times a second
- Leverage Code like a serverless architecture
- Author services that are designed to run quickly and be responsive — They are not for batch work
- Ensure that all execution paths result in a resp.success or resp.error call
- When starting a service — the most efficient way to instantiate the ClearBlade object is using the ClearBlade.init(request:req) This call does not require any interaction with the backend but instead simply configures the ClearBlade library to use the existing valid auth token
- Add descriptions that identify under what conditions this service is known to be called
Performance
- Be careful when pulling back large datasets. Since the dataset has to be parsed by the code service, it is sometimes more efficient to make 10 calls of 1Mb payloads rather than 1 call of 10Mb
- Calls to third party software can fail; handle those slow and failed response gracefully
- Be careful of too much logging. If your service is running at production scale, don’t needlessly log. Turn off your logs
Reuse
- Identify common calls early and make reusable libraries. This will minimize both development and maintenance time.
- Keep libraries focused. Each library will be executed when the microservice runs, so make sure you only are bringing code you need into scope
Debugging
- Debug services at runtime by dynamically turning logging on and off for the service as needed
- Check your failed service log to see which service didn’t execute correctly. You can always fix your code and then rerun those service calls to recover state.
Messaging
Basics
- JSON has become a very friendly structure for IoT. Passing strings of JSON is an easy way to quickly parse and work with your data.
- Topic patterns are a powerful way to structure and optimize communication. Use them to your advantage http://tinkerman.cat/mqtt-topic-naming-convention/
- Messaging can be treated as a topic or as a queue. Identify the pattern that is right for your system and workload
Data
Basics
- When integrating with third party software — avoid duplicate table writes. When using ClearBlade attempt to pick a single location for data structures rather than reproducing them across different databases. At scale 3 writes to different connections can be a significant workload
- The default timestamp pattern is 1970–01–01T00:00:00Z
Connections
- ClearBlade generally behaves as expected when tables have primary keys. Logic to handle third party databases without PK’s can result in unexpected results
- Not all databases are optimized for certain activities (IE — Cassandra is slow for reads) Make sure you are using your other databases in ways that are optimal
Triggers
- Avoid looping triggers. Trigger execution can result in activities which in turn cause more triggers to fire. Be careful to not create an infinite loop of triggers that will consume your resources
- Leverage the req object to identify state. The req object brings key information about the reasons for and under what authority the trigger was fired. Use this information to optimize your trigger logic and cut down on total execution time.
- In systems that have a high volume number of messages, don’t trigger on each message but instead use a repeating timer that pulls the message history (queue) and then clears it. This will allow you to optimize your CPU and interactions.
Timers
- Use dynamic timers to make sure certain events have happened. If you want to make sure a device communicates every 30 minutes, then create a dynamic timer. Each time you see the device move, reset the dynamic timer in your service. Unlike normal timers, recurring timers only run when they need to
- Timers recurring in under ~1 sec duration can become un-ordered to the horizontal asynchronous behavior in the platform.
Users
- When doing third party integrations, store tokens and keys for those service in the user record. This will let you secure access to those tokens by default for the user.
- When integrating with a third party registry, Create an anonymous service that will check the external registry and then upon success perform the auth in the ClearBlade service. This will let you grant ClearBlade tokens based on the third party system
- Verify your email addresses
- Downcase your email addresses at creation. The specification for email is nonstandardized. Protect your users from creating duplicate users by enforcing all lower case characters.
- Rapidly build 2-factor auth with the use of anonymous services
Devices
- Consider a portal for defining a common device experience. This portal will allow you to standardize what devices and schemas get submitting rights into your system.
- Device can have completely unique schemas.
- Provisioning devices into the table without enabling them. This will let you create the device record but not allow it to interact until the device is officially registered in the field.
Scalability
- When attempting to perform large amounts of analytics, consider raking and grooming your data at the edge first. The edge will let you process sensor data before it goes to the internet and incurs cloud costs.
- Consider the right data storage for your goals. Some cloud services like Splunk have a hard time keeping up with ClearBlade mqtt. If the end service cannot handle the data volume, try a different strategy that does not cause blocking behavior in your services.
Scalability
- Ensure adapters handle token expiration and reauth
- Maintain a thread for each unique device that is connected via the adapter for making sure it’s using its correct credential
- Consider automating the provisioning process as part of your adapter spec so that new devices can quickly provision in an abstraction fashion