An idea blooms

There’s all sorts of problems out there to solve but you have to be careful.  Not everything can be fixed with a microcontroller.  Sometimes it seems like it’d be nice but then you hit hurdles – oops, now I need wireless.  Oops, there isn’t a battery big enough to make this work.  Oops, this is physically impossible.  The truth of the matter is that there are few problems that can be fixed cleanly with a microcontroller – and I’m all about clean.  I hate kludgy systems grafted on to other systems grafted on to Twitter.  Sometimes you just need to wait for a problem that is begging to have code written to solve it.

And I found that problem.  Here it is:

That wasn't like that before!

I know what you’re thinking – who could be responsible for this heinous misuse of the trash can?  WHO WOULD DARE?!  WHAT KIND OF ANIMAL WOULD DO SUCH A THING!?  As luck would have it, we know exactly who the offender is.  Here she is doing her hardest to appease us with cuteness:

 

Bad dog! Bad adorable dog!

Really, what kind of fool does this dog take me for?  That’s only a half-windsor!  Not bad for a creature without thumbs, I’ll admit, but this does not excuse her bad behavior. Not at all.

Naturally she only does it when we’re not in the house.  And it doesn’t take long at all – this isn’t a desperate gamble to get our attention, to force us to recognize that we are abandoning her and she has to take measures to fend for herself.  No, this is after perhaps a half hour of being alone.  That’s apparently all the time it takes her to realize that there’s bacon grease in the trash.  And boy does she love her bacon grease.  

What to do?  Hide the trash when we leave?  Not bloody likely – that sounds like work.  Get a different trash can – one that closes tightly and allows no dog to access its contents?  That’s also work – I hate going to Bed Bath and Beyond.  Plus, neither of those ideas trains our dog to be better.  Who knows where all that energy will be directed if not the trash – perhaps to the kitty litter?  I want to train her so she leaves things alone, not accommodate her bad behavior.  But she’s crafty – she only gets into the trash when we’re not present.  So how persuade her to stay away from the trash when you’re not there?  This is a solution that begs for an automated approach!  What are our options?

Well, because I keep up on the latest in tools from Cool Tools I have the inside track on a neat little device used to train cats:  the Innotek SSSCAT Cat Training Aid  It’s basically a can of compressed air with a motion sensor on it.  You put it wherever you don’t want your cat to be – usually up on counters or something like that.  When the cat jumps up there and gets within range of the motion sensor it releases a burst of compressed air that scares the crap out of the cat.  People say it works rather well – it’s a consistent negative association for the cat.  Best of all is that it’s automatic – it doesn’t rely on a person being there and consistently disciplining the cat.  That sort of consistent training helps the lessons stick.  If I want to train our dog to not get into the trash I need something like the SSSCAT.  I could probably use the SSSCAT – it wouldn’t be a bad idea for me to own one anyhow since we have two annoying cats.  But damnit, it’s spring – when a young man’s fancy turns to… engineering.  Of course.  I want to make something  and this is perfect for several reasons.

First, it’s not too complicated.  I’ve thought it out and it will only need one or two sensors as inputs and one noise maker as an output.  There are no complicated communication interfaces, no zany analog front ends, and no high frequencies.  You could code the whole thing in assembly if you wanted.   Second, nothing exactly like what I need currently exists.  The SSSCAT might work – perhaps I could place it behind the trash can or inside of it somehow, but that might just as easily not at all work.   Third is that I can add features that aren’t in any product I can find.  Inventing something that doesn’t exist but would make your life easier is the reason I got into this engineering racket.  Some people write to make the world a better place, some enter politics.  I find new ways to be lazy. Someday I dream of a world where everyone can be lazy.

So it’s decided – I’ll save the world by making something that yells at my dog when she knocks over the trash.  It’s a tall order, but I think I’m up for it. Next time I’ll discuss high-level design: why it’s important, what sort of documents you should produce as part of it and what kind of information they should contain.  We’ll also discuss project organization and folder structure if you’re lucky. 

An In-Depth Look

I had someone ask to share the loops I was talking about in the earlier post for analysis.  Ask and ye shall receive!  Let's start with the loop that performed awfully.

Stare into the belly of the beast...

Here's a quick walkthrough (roughly from left to right):

1. Load a text file as a spreadsheet with a break on all newline characters - this gives me all of the lines in the file in an array of strings (there's probably an easier way but this was most direct at the time.
2. The array of strings is fed into the first FOR loop.  The loop is auto-indexed on the array of lines.
3. Within the FOR loop there's a case statement that handles different lines in the text file differently.  There's header rows and such at the beginning so basically the case statement allows the loop to ignore those and start with the data rows.  
4. Inside the case statement there's a second FOR loop - this one loops over each field in the line.  It is indexed by the array of integers.  Each row of that integer constant contains offsets and lengths for a string subset call. (More on this in a second)
5. Inside the second FOR loop there's a case statement that has a case for each field.  Every field except the last is handled the same.
6. Inside the case statement there's a string subset command fed with values from the constant integer array.  These values provide offset and length for each string subset call except for the last.  The last is the special case mentioned above - there's no length associated with the last call to string subset (not wiring the length causes it to read until the end of the line).
7. When each field is picked off, that text replaces one of the elements in the constant string array being fed into the loops near the bottom.  The string array has an element for each field in each line, and the integer constant contains the array element within that string array to replace with the current field.
8. When a full line has been read and turned into an array of strings, that array of strings is appended to the list of lines already read.  
9. The loops then continue with the next line.

 I hope most of you can already guess what the main problem is so far.  It's the 'this loop inside this loop inside this case inside this loop inside....' yada yada ya. That's bad - for speed anyhow.  At first I tend to try to make my code.. shall we say, robust?  Able to handle even impossible situations?  I also try to make it clear, and clear means verbose or at least, doing much more than you need to to promote uniformity and centralization of data.  This often leads to unreasonable performance through over-engineering.

Let me give you an example -  I'll bet something like this has happened to you.  Let's say you've got a program that communicates with another system.  For what you're programming, you have to change your behavior depending on what commands the other system sends you. So as a hacker, you start hacking.  You read the first command you need to implement:

"Set mode to passive - in this mode your system will only respond to messages sent to it by the external system."  

Ah, easy!  I'll just make a boolean that tells me whether I've received that message!  Then I can read that boolean and determine whether or not to send messages automatically.  Great.  Second message:

"Set mode to active - in this mode your system will automatically send the following messages at these rates:..." 

Perfect - when you receive this just toggle the same boolean from before to set yourself back to automatic mode.  Third message:

"Set response delay to minimum - when you receive this message delay all automatically-sent messages by one frame."  

Okay... so I'll make a boolean for this one too?

"Set response delay to maximum: When you receive this message delay all automatic messages by three frames." 

Okay, okay, no, wait, I should make that last one an integer or something and not a boolean, I'll just go back and change that...

And right about here my bat-sense starts tingling.  "You're going to be doing this forever!  Use a state machine!  Otherwise you'll have a million variables and you won't be able to keep track of what combination of booleans and integers defines what state and what behavior!  STATE MACHINE!"  And a state machine sounds like a good idea - it allows you to be more rote, more design-oriented, more organized and more certain that your system works.  Implemented properly they improve deterministic behavior in complex systems.  So I add a state machine and start defining all the states I'll need: automatic, passive, umm, delay?  No, so far it's just two states but I'm sure there will be more.  Let's read the next requirement:

"....."

Ah, so we're done are we?  So I rigged up a whole state machine architecture when all I really needed was one boolean and one integer.  Sure, a state machine will WORK for this but it's certainly not efficient.  It needs memory, time, special coding practices, etc.  A boolean will do the same job much more efficiently.

And that's my problem - that's why I build loops inside of cases inside of loops (yo dawg, I herd u liek loops....). I think I'm doing something to make my programs deterministic and robust.  I'm actually making my programs robust, deterministic snails.  Like a giant snail-tank that's crushes anything in its path and is always on-time.  I don't think anyone wants that.  Unless that's what you ride to work every... month?

After much profiling of various methods of implementing this functionality, I settled on the below implementation:

So fresh and so clean!

Yeah - look at that.  ONE LOOP!  Seven calls to string subset (running in parallel) and no Build Array blocks!  Loading a 48000 line file used to take an hour.  Now it takes two seconds.  So how did I manage to rid myself of all of that cruft?

1. First, I stripped off the header rows from the array containing each line.  That removed the need for a case statement inside of the first FOR loop (the one that ignored the header rows).  
2. Then I removed the inner loop that looped over each field and just put all of the string subset calls in the same place.  This allows them to run in parallel instead of serially (if Labview is as smart at figuring out how to parallelize things as is claimed).
3. Since I'm not ignoring any rows any more I can replace the Build Array block with a simple auto-indexer on the output of the FOR loop.  This significantly sped things up. 
4. And finally, I configured the loop for parallel operation.  To do this right click the FOR loop and choose 'Configure loop parallelism' or something like that.  This lets different iterations of the loop run at the same time.  If you think about it it makes sense.  As long as the next loop iteration doesn't depend on the last loop iteration then you can just run them all at the same time.  If I had 48,000 processors then I could have each of them process one line of text and have this job done in microseconds.  When you use it though, pick a small single-digit number for the number of parallel loops. There is overhead associated with managing multiple threads so increasing the number of parallel threads starts to backfire after a while.

The second block diagram is much better programming than the first simply because it's quicker.  It's like I always say: any automated process is better than any manual one because it's consistent AND because once it works you can speed it up later.  There's no way this program would have been useful to me if it took an hour to load one log file.  So don't overcomplicate things!

Boo Labview!

So I'm not posting about the microcontroller stuff I swore I'd start last time. Well, this time I swear there's half a post sitting on Blogger ready to be finished someday. That one IS about microcontrollers. This one's about Labview - mostly because I had a fun little time with it today.

We have a customer who has designed some rather interesting test equipment utilizing Labview Real-Time. This means that for months I've been working with Labview for all sorts of fun things. Since their test equipment is all National Instruments-based we've decided to double down and use TestStand to automate the testing that utilizes their equipment. This means that one of the fun things I invented to help our workflow along was a piece of Labview that reads in a CSV file containing descriptions of what inputs to apply to various VIs and what outputs to expect from them and turns it into a TestStand-compatible sequence. This has saved us more time than I can imagine - mostly because we didn't even try to write any of the sequences by hand. I was working with someone who works for our customer and he actually attempted to write a 500 steps sequence by hand. I don't think he got even halfway through. Keep in mind this is a rather tedious set of steps. Picture something along the lines of "Press '1' on the computer keypad. Verify that '1' appears on the computer display. Record the result. Now press '2' on the computer keypad. Verify purple monkey dishwasher. Hit self in head with hammer. Melons."

You see? I couldn't even get through two steps without losing my mind and desiring to physically harm myself. When you use a CSV file you can search and replace. And copy and paste. Copy step 1, paste to step 2, search and replace 1 with 2. Presto! Extra nerd cred if you use regular expressions! Don't hire a person to do a computer's job. Use the right tools!

In the spirit of using the right tools I wrote a lot of Labview today. You see, our customer gave their test equipment an option to log a LOT of CAN bus traffic. The only problem is that due to the (fast) way they write the log you can't be certain that everything is in order time-wise. I don't blame them - it's a real-time system. You do what's fast and leave the post-processing for a bigger computer (though do note an irony - I'm fairly certain that the real-time PC is more powerful than the host PC it's 'slaved' to). They suggested a rather cumbersome process to import the data into Excel and sort it but I say nay sir, nay! I hate things with more than one step. That's why I have an iPad - only one button. There's no question about what to do. The iPad is off? Step 1: push the button. Success! You'd be surprised how often it works.

I wasn't happy about eight steps and I knew that in the future we would want to use this log to verify behavior of the system under test, so I decided to write Labview to read the log, organize it, search for certain messages, judge skew, rate, etc. So it made sense to write a Labview program to parse the log, organize it, search it and collect statistics. Now keep in mind that this log is distinctly not... how shall I say? It is distinctly not brief. The log for a 10 minute test is 20 megs. And it's all text. No pictures of cats whatsoever (not that they wouldn't improve things). Admit it - you're a little surprised that there can be that much text and that I can be interested in so much of it. But hey, we're not barbarians. We can handle this much text. It's no problem for a computer - especially the amazing computers we have today! Computers are information processing MACHINES. And a machine means I don't have to do the work for myself. That means less thinking. That are good.

And all it takes is....

Um...

Theoretically it will take around six hours.

Yeah, yeah, I know - that sucks. That's terrible. I should feel ashamed. *I* wrote something that take six hours to process 20 megs of TEXT? 20 megs of video is, what five minutes maybe? Naturally this depends on quality, but let's face it - you'll blow through 20 megs of high-definition video in much less than an HOUR let alone SIX. And I wrote a program that couldn't handle 20 megs of text in less than 75% of my work day.

Hey, I like getting paid as much as the next guy but even I get bored. Plus, I want to get this log analyzing done quickly so I can do more of it and be DONE with everything quicker. I want to respond to our customers questions in minutes when they expect the answer tomorrow. I like to impress, I like to excel and what I've done here is not it at all.

So I start profiling. So I load up a 5 meg text file (one of four in a typical log set). How long does that take? Milliseconds. Single-digits. Not worth mentioning. That is not the problem. I sort the data by timestamp - sounds intensive right? I have no idea what Labview's built-in sorting functions are like. Quicksort? Bubble sort? No idea. Could be a lot of room for improvement there. But... nope. Very brief - a small amount of the total time. Rather impressive actually. So I break it down; I limit the number of lines so I only have a thousand. Now the whole process takes a second and a half. I know it's not loading the file, I know it's not sorting the file so it must be loading all of the data line-by-line into a 2D array. One big ugly FOR loop in the middle of my VI that is slowing everything down. And no matter how much I profile, no matter how much I change the loop always seems to take about half a millisecond per line. My first attempt used one string subset block but it was inside a second FOR loop that ran once for each field in the line. Surely a FOR loop inside of another FOR loop significantly slows things down. After all, they keep telling me Labview is inherently parallel. Multi-processor, hyper-threaded blah blah blah. Putting that FOR loop inside the other one forces a serial process. If I used eight string subset blocks at once then it could thread it any way it pleased. That should help shouldn't it? No dice. No change. Ah, look! I'm indexing an array - if I can find a way to remove that from inside the loop.... no help. What if instead of string subset I used one Scan Into String block? That might.... provide no benefit at all.

I suppose if nothing else I could remove this Build Array block and see if I can't just let the FOR loop auto-index the output but I doubt that would....

....

....

Each iteration of the loop now takes 6 microseconds. Down from 500.

This is why you can't trust Labview. This is why Labview Real-Time is an oxymoron. It does not encourage nor make easy optimization. Now, I will be the first to admit that I created - from scratch in one day - a program to process and analyze tens of megs-worth of text data and display it in a user-friendly way. This program will make me more efficient and productive and I have Labview to thank for that. And I do.

But Labview just feels like a toy to me now. When you have ten lines of text data who cares about optimization? When you have a hundred it's a question of how much do you care about thirty seconds of watching a progress bar. If it's a thousand you go and get a coffee and when you come back it's done.

But if it's half a million? You can't just say 'Well some things take a day to run.' That's silly. That's wasteful. Worse - it's unprofessional. Wall Street trading firms have gone so far as to implement their risk analysis algorithms on FPGAs!. It used to take them the better part of day to process their data. Now? Something like fifteen minutes. And guess what? They win. Their competition still takes hours. Now they can act quicker or make better algorithms to work smarter. Better efficiency gives them more options. That's what a professional does. A professional does not shrug his/her shoulders and say 'I dunno. Let's just buy a faster computer.'

So I can't trust Labview. I moved all sorts of blocks around trying to figure out which one was the culprit. They all looked the same. I had no idea what was under the hood. Was it linear complexity or exponential? National Instruments won't let me look at the code. As far as I know they haven't given me the tools to thoroughly analyze their entire architecture. Where's my gprof report? And was their Build Array block seriously doing something as absurd as copying a ten thousand element array each time it wanted to append one more line of text?

If I had done it in C I could have utilized pointers and custom data structures to make a faster program. If I had done it in C I doubt it would have loaded the file in more than ten seconds to BEGIN with. Only in Labview are you encouraged to just plop down blocks without any regard for the cost in time or resources. Even if you cared there's no way to know other than tedious experience who the culprit in your bad algorithm is this time. Does that addition block break down into one assembly command or 100? You will never ever know with Labview. That makes it a toy. That's why you can't trust it.

Edit: I realize that some people (among the many I'm sure who read my blog) will wish to argue with me and say that the problem I've outlined above with respect to Labview is a problem with EVERY computer language. Seasoned programmers know that you can't just call any function willy-nilly and assume it will be fast. I agree with this. It can happen in C - should I call printf in each iteration of a loop, or build a buffer and then print it afterwards? Building the buffer will be much faster than a function call which will then significantly speed up your loop. And this is, of course, not only something you need to keep track of in Labview but in C or Python or any other computer programming language I can think of.

A seasoned programmer also knows that you adapt your approach to each unique problem you encounter. C provides you a nice framework to solve your own problems which includes pointers, function pointers, structs, a myriad of different loops, bit shifting and if none of these are sufficient you can use assembly. Nothing other than rolling your own co-processor with VHDL is faster than assembly. But with Labview you don't have these options. Pointers? Don't exist. Game over. That's your number one tool for optimizing code. For example, the block I had a problem with was build array with a 2D array. Essentially, I have an array in which each element represents a line of text with several fields. The line of text with fields is stored as an array of strings (which, if you want to get into things, are ultimately arrays of characters, null-terminated). That's a lot of arrays but the easiest way to imagine it is that I have an array where each element store several fields of information. What I would do in the innermost loop was parse each line of text to read the fields and shove it into a data structure and then append it to the array with all the data I had already collected.

Well, it seems the Build Array block was to blame. The only way I can explain the terrible slowdown is that somehow Labview stores its arrays like strings - there's one pointer to mark the start of the array and where the array stops there's a special terminator entry. Just like a null-terminated string. So when I wanted to append the latest data to the array, I think it iterated through the whole array until it found the terminator entry and then appended the data there. For small arrays this isn't too bad, but I was getting to 48000 lines per text file (with four text files worth of data). Iterating through that much data every time you want to add another line gets expensive. And it was probably also re-allocating memory for the array each time too! In C I would at least have done the following:

1. Allocate enough memory to hold all of the file data before I start the loop. I might count all of the lines in each file but that might take too long. It'd be better just to grab then length of each file and estimate what I needed.
2. Maintain a pointer to the last element of the array rather than have to iterate through the array each time I wanted to add something. It will not use that much memory and it significantly speeds up.
3. Don't use another loop to process each element in each line - unnecessary loops introduce unnecessary overhead.

Those are the only ones I can think of right now (and #3 is kind of a stretch) but I already know that with those ideas I could make a much more efficient loop than Labview. If I had all that control then I could write something really fast. With Labview I simply don't. I have no idea how any of their blocks work or even how the auto-indexing on a FOR loop works. All I have to work with are the tools they give me and I have no indication how any of them are different.

Many people will say that it's good I'm thinking about Labview in this way - no programming language is perfect, so learn its quirks and work within them. The only problem is that's not how National Instruments wants us to think of Labview. We can't judge it like we judge C because it's not meant to be C. It's supposed to be a magical language that frees us from worrying about the code and cycles and memory and instead focus on our ideas, our algorithms. Labivew, they assure us, will free us from having to think about how to efficiently implement our algorithms. They say anyone can be productive with Labview. I will give them this: almost anyone can use Labview and that is impressive. I've seen Labview created by Computer Scientists, Physicists, Electrical Engineers, Technicians, Chemists - you name it. And most of it works just fine for what they want it to. But it's generally badly written and inefficient. In some ways Labview makes it much easier to be badly written and inefficient - specifically because it essentially works so hard to figure out what you mean instead of what you said. From the viewpoint of someone who's picky about code that's despicable. From the viewpoint of someone who wants computers to make peoples lives better, that's laudable.

Beyond Arduino

I own an Arduino. And why not? I've always liked the AVR - mostly because Atmel has always had a good focus on hobbyist-level tools. WinAVR is an excellent free toolchain and can interoperate with a wide number of IDEs due to the way it mimics the functionality of the GCC toolchain (THE standard C compiler). The Dragon is an extremely cheap ISP with debug functionality and HV/Parallel programming interfaces for when you screw up your fuses and it can program nearly all chips in the AVR line. The AVR architecture is similarly impressive due to its high MIPS/MHz ratio and hardware multiply functionality.

Wow, that was in-depth, I hope I didn't lose anyone. Hey, vast majority of Arduino users: did you get any of that? I'm sorry, I'm sorry, I don't mean to be a jerk. No, the vast majority of Arduino users don't understand that. But that's okay. I'm an electrical engineer - trained and all! I've been programming microcontrollers for, oh.... Oh God - eight years now. I think I might be... old.

So I'm old, and I'm an electrical engineer. That's why I know these things - to me it matters. If an AVR is a bad choice for a project I have to know why so I will be able to tell whether to replace it with a PIC, a 68K or ARM Cortex-M3. Most people who use the Arduino are not electrical engineers, develop embedded software for a living or necessarily have ever programmed before. Now, despite the fact that this causes some pretty hair-pulling questions on Chiphacker (I'll bet that question is gone by the time I post) it's a good thing. Embedded systems have the potential to make our world a much smarter and better place in the hands of those with the right knowledge and world-view. I mean, the most basic application I can think of would be intelligent power sockets for everything - track usage of power, schedule things, auto-turn off of dangerous items, etc. You could make your house more efficient (power-wise) and safer (auto turn-off on a curling iron? Yes please!). Someday, the halting, uncertain steps taken by people who have never soldered before will lead to a better world through practical software.

Which, to me, makes it all the more important that if you're going to have a good idea and succeed at making it ubiquitous you ought to know a thing or two about writing safe, deterministic software that will last forever and gracefully handle all errors.

This is just as hard as it sounds. But worth every bit.

You've seen it just as much as I have. I mean, look at the gas station - the electronic pumps that take your credit card. First off, they beep as if they have something to say and then it takes them three seconds to start to update the display. And then it updates a little bit faster than molasses flows. So it asks you for your zip code. You punch it in the numbers and every time you hit a key you hear the beep beeps but don't see the numbers show up. No, for some reason that takes about 10 seconds per digit. So if you mess up you need thirty seconds or so to fix it. But that's only a little annoying - try this on for size: has your dishwasher ever crashed? Would it refuse to clean anything and spit out an error code you can't look up because you don't have the service manual? Did unplugging it and plugging it back in fix it? Do you know how many lines of code are in your car? Did you know I have a microprocessor in my PEN?

While it's true that anything remotely dangerous with software in it is (generally) tested to some degree to make sure it doesn't kill you we have to ask ourselves - is the shoddy software we see in something so simple as a fuel pump with a display and a keypad something we want to put up with in (eventually) every single device we own? What is so frustrating is that we know how to write software. We know how to read keypads, we know how to write numbers and letters on a display. We know how to dial a phone and ask a remote system whether that credit card number is valid. And if people would just write software correctly we could avoid creating a million devices that need to have their batteries removed every day due to blue screens.

When I say 'we' know how to write this software I mean... I don't quite know who I mean. Dudes at NASA who couldn't rely on taking out batteries because the batteries were IN SPACE! Those dudes knew how to write software that worked - period. The stuff in cars, airplanes, medical devices is not perfect but as a friend of mine would say it's not a cell phone either. The quality is simply higher because otherwise people die. Right now, that sort of expertise costs money - you need the right people to design AND test your software so that it performs to the standards of whatever certifying body you need to satisfy. If there aren't any standards you need to satisfy then you don't bother - you hire someone who grew up on VBScript and PHP instead of someone who grew up on assembly language and eating rusty nails for breakfast.

But the point is that the expertise is out there; the information is out there. Our collective knowledge knows how to write good software, so the goal should be to get the information out there and make it easy to use.

So, back to the original point of this post. What does this have to do with Arduino? Well I'm sad to say that while Arduino is great it's pretty basic. It is somewhat brain-dead: it encourages blocking calls (ie, idling in a do-nothing loop until something happens) rather than use asynchronous interrupts. It skips over more advanced topics that form the basis of some of the techniques which ensure safe software such as callbacks/function pointers. It shuns even basic lessons such as parts selection (newsflash: you're going to use an Arduino. You will buy a shield - you don't need to know what's on it). You're certainly not going to learn anything about design for manufacturability (unless you're making a shield). Ever heard of a state machine? Circular buffer? Process stack pointer vs. machine stack pointer? All very important!

The Arduino is a good place to start, but if you're serious about making something worthwhile that has software in it, your attitude can't be 'I don't care about good correctness, suitability or efficiency. I just care if I can get it to run'. This attitude leads to terrible software and terrible products to boot. It can be avoided. Over the course of... the rest of my life I intend to write a series of articles/guides that instruct people on what the essentials of writing good software/good overall design are and how to apply them. This will involve every aspect of the process I can think of from microcontroller selection to getting a PCB made. I'm going to write the articles, publish the code, open-source the development kits and take a lot of pictures. Yes, there may be selling of things involved, but you only have to buy them if you're lazy. All of the tools (as far as possible) will be free to use and open source and all of the design files will be made available with an open-source license.

If you want your software to be faster, more responsive, more efficient and just overall better, listen for a while. Even if I don't teach you something I may point you in the right direction.

Labview is Actually Not All That Bad

In stark contrast to the flurry of epithets I hurl at Labview on a daily basis at work I will now say good things about it - because it deserves it!

The blocks are actually pretty old school. If you get stuck trying to decode a string in Labview your tools look surprisingly like what you'd expect from the C standard library. Granted - you don't need to worry about NULL termination or anything, but most of the basic building blocks (like string subset, search and replace, etc) promote code that acts (in form) like old-style C code. It's not like writing the routines yourself (iterating through each character, comparing, discarding, buffering, etc.) but the solution you come up with to manipulate strings will probably feel like an old-style C solution does. I also like the way the control structures work - you get nice features like shift registers (that teach you about instantiating a variable before the loop) or straight-through tunnels that just take the last value from the loop and pass it out. You can make complicated while or for loops with conditional terminations or conditional continuances. These are very welcome forms to have present. There's a lot of nice tools and abilities in Labview that don't feel new and gimmicky, but instead old and tried.

Second, while most Labview VIs are just a mess plain and simple what I've found is that a messy VI usually means messy badly-organized code. A clean, well-organized VI means you've created sub-VIs in the right places, organized the right data into clusters, used the right type of loop, etc. If your VI is clean and organized, chances are your code is too. Thus, by seeking to visually clean up your block diagram you can actually write good code. Of course, the opposite is not true: you can have great code that still looks like a mess. Because you just can't get around that in Labview sometimes. But chances are if the block diagram looks good the reason is because of a highly-organized coding mind behind it. It helps people learn how to organize code by presenting it as a visual problem instead of an esoteric abstract one - and many people are simply visual learners.

Third, I'm ecstatic as hell that whenever I Google anything related to Labview I come up with an answer - period. Someone has tried to do what I'm doing or for some insane reason National Instruments make a guide on how to manipulate TestStand sequences programmatically from Labview. Quite simply put I spent hours trying to figure out a problem on my own and when I get the bright idea to Google it the answers is invariably THERE. Done in five minutes. At this point in my career I'm a little more focused on results than banging my head against a wall to 'learn' so that feels really good. I have no idea why all of this support is out there but it is and it makes me happy.

Fourth, Labview is actually free of a lot of the object-oriented crap that plagues many trendy languages today. Yes, Labview is actually a bit trendy by itself, but (and this impresses me because the more I think about it the more I realize it's true) Labview is actually pretty old-school. Some old-school hard core stuff you can't do (like function pointers - but then only kinda) but I'm pleased that I haven't seen the term 'inheritance' once when dealing with Labview. True, all the things that make me hate C++ might be in there but I haven't been forced to deal with it and I haven't seen anyone else's work that deals with it either.

Don't get me wrong - I still find plenty to hate about it (and I may get to that later) but the more I consider it Labview feels like C but graphical. And that's not terrible.

Good Coding Practices #...?

I have a semi-ongoing series in good coding practices... to the extent that I ever update my blog anyhow. Lately I've taken up work on an iTunes Android Sync tool. Is anyone else amazed that there are very few good programs that will sync iTunes playlists to an Android device? There are some out there, certainly but for some reason each of them tends to have one or two major flaws: way too slow, randomly renames your songs to the titles of different songs, costs money - the usual complaints. So I looked into it and it turns out that iTunes maintains an XML version of its library information file. But then I discovered that the Android playlists are stored in a highly technical format called ASCII-encoded text files. Let me tell you, it took forever to crack that puppy.

Well when all you have is a hammer every problem looks like a nail. When you have Python every problem looks... easy. So I decided to make a Python program to:

1) Read the XML library file.
2) Figure out what playlists are in there
3) Figure out what songs are in those playlists
4) Figure out where those songs are
5) Make Android playlists from the iTunes Playlists
6) Copy the playlists and music files to the Android device
7) DANCE!

So at first I tried using my favorite parser - SGML parser. But it turned out that SGML parser doesn't handle truncated tags. You know - the ones with nothing in them? With only a start tag that has a / in it and then it's done? Yeah, those. So I had to switch to expat which isn't so bad either.

But enough of that! I'm going to show you what I did that's a good coding practice. The iTunes XML file has several parts in it: a general section that describes the library, a tracks section that describes each track and assigns it a unique ID, a playlists section that describes the playlist and lists the unique track IDs in the playlist.

I wanted to start off by parsing all the goodness of the general library section and ignore the rest while at the same time planning ahead so I would... be able to figure out where to put the code to parse the rest of it as well. To that end I present a random code snippet:

def handle_data(self,text):

if self.current_tag == KEY_TAG:
self.current_key = str(text)
print "Key: " + text
elif self.current_tag == INTEGER_TAG or self.current_tag == STRING_TAG or self.current_tag == DATE_TAG or self.current_tag==TRUE_TAG:
if self.current_parent == LIBRARY_KEY:
if self.current_key in libraryKeyList:
print self.current_tag + "=" + text
self.tempDict[self.current_key] = text

elif self.current_parent == TRACKS_KEY:
pass
elif self.current_parent == PLAYLISTS_KEY:
pass
elif self.current_parent == TRACK_KEY:
pass
elif self.current_parent == PLAYLIST_KEY:
pass
else:
pass

self.current_key = ""

Some explanation: this function handles data inside of tags. It handles key tags specially, but handles tags that contain data (integer, string, date, etc) differently still depending on which section they reside in. So you can see I've written the code that handles the data in the library section but left out handling data in all the other sections. But this is by design: if I wasn't planning ahead I wouldn't have put the if statement that checks what the parent is in that function. I would just have put the code that handles data for the library section without verifying that I was still in the library section - and then it would have handled a whole lot of data in the rest of the file.

By putting the parent key check in there and explicitly listing the different situations that I want to code for I'm doing two things. First, I'm specifying the exact situation I expect this code to run in - putting my assumptions right out there in the code. Second I specify all of the other situations that I haven't yet coded for but want to in the future. I'm using the code to inform myself (in the future) that I need to put code there that does something different. That's the good coding standard.

It can be used in a variety of languages. In Python use the above form but make sure that you put the pass statement in an empty case - otherwise it gets angry. In C you can use #warning directives to produce a warning when you know you'll have to write some code but just haven't yet. Like '#warning Will Robinson, you didn't handle the default case!'

On Excellence

I fancy myself pretty good at C. Not great, but pretty good. I can find my way around source code, I can write from scratch, I can debug with the most average of them. I'm handy in a variety of ways and I eat source code for breakfast.

This didn't happen in college. This didn't happen in high school. This didn't happen in freaking grade school. I have been programming since I was 6. I started off with AppleSoft BASIC on an Apple IIc knockoff (Laser 128c was the correct answer for those of you playing the home game). After AppleSoft BASIC it was GW-BASIC on the 8088 and QuickBASIC on the 286 and up. But roundabouts high school I decided I had to learn C - because that was the language that grownups used. So I bought a C book (the right one as it turns out - if you want to learn C get this book first), downloaded DJGPP and got to work!

I'm having a computer weekend (putting a computer whose hard drive failed back into working order) so I'm going through old files looking for utilities and just plain reminiscing. I decided to see what my old C code looked like.

Oh God, it's awful.

It's terrible. Here's an example (with some helpful comments from future Steve):

i=-1;

fseek (readfile,0,SEEK_END); //Set starting point to end
size = ftell(readfile); //Find file size
fseek (readfile,0,SEEK_SET); //Set starting point to start

//FS - Seriously? Is there no better way of getting the size of a file?

//FS - Oh god, who gave me malloc?

readfiledata = malloc(size); //Allocate memory for char

printf("Filename is %s\n", argv[1]);
printf("Size is %d\n", size);
printf("Copied %s to steve.tmp\n", argv[1]);

//FS - WHAT?! WHAT?! Index of -1?

i=-1;

do
{
//FS - SERIOUSLY!? Is there no easier way to get all the data in an array? Did you not look?

readfiledata[i] = fgetc(readfile); //FS - OH GOD YOU ACCESSED INDEX -1 OF AN ARRAY!
i++;
}
while(!feof(readfile));

In case you were wondering, I used malloc() and, no, there is no corresponding free() call. I relied on the fact that the OS would free the memory once the program exited. There were variables defined in .h files (no, not extern defines, plain old defines). There were what should have been arrays of constant strings were 'initialized' using sprintf (copy constant to string) rather than just initializing them when the array was defined (as any normal person would do).

And the best part is that the whole program I made basically amounted to a regular expressions parser. All I needed to do was remove images and other formatting from HTML files so they'd be easier to print off and use less ink. That could have been done a lot easier.

This was 1999. So, becoming average takes at least 12 years of constant use of a skill - and I still screw up. Looking at this I can see a lot of myself in new grads coming out of college - the same mistakes, the same assumptions, the same basic design assumptions that end up making bad code (even if it runs). It falls in line with Malcom Gladwell as he writes in Outliers: if you do something for 10,000 hours you'll be great at it. It's not necessarily innate skill, it's practice, practice, practice. It's why I'm a professional programmer and not a professional trombone player - I just code a whole lot more than I play trombone.

So knowing this I can see where new grads come from - hey, they haven't had 10,000 hours of programming, they probably haven't had 10,000 hours of anything engineering related from their college experience. 10,000 hours is 3.5 years at 8 hours a day and that's just for one skill. Engineering is a whole plethora and if you don't know where your career is taking you, why bother practicing one skill over another?

That's my strawman argument - I don't agree with it. My question is - if you know you won't have you 10,000 hours in whatever you want to be good at by the time you graduate college (and by extension, be able to show some really awesome work to a prospective employer) why didn't you start earlier? Did you plan to be average? To be right in the middle of the pack, to not stand out? To be, essentially, replaceable by any other member of your graduating class? Did you plan to go out into the job market and have a big corporation tell you what you should be good at instead of deciding it yourself? Didn't you get into engineering for a reason?

I see people on both sides of this question and you can tell them apart right away in an interview. The people who don't know why they're engineers - the ones who didn't start early excelling at something they loved and wanted to do - come to a job interview and basically want you to tell them what kind of career they should have. They hit the middle of the road for all of their classes - probably picked whatever electives were the most popular because they didn't really care about the difference, didn't have an opinion on what they wanted, didn't find anything particularly exciting and just followed their friends into a class. Their senior projects were whatever they were assigned and they just sort of did them. They don't speak about them with passion, they just wanted to graduate and they needed a project, so they did it. And they're not dumb - a lot of them have 4.0 GPAs for what its worth. But since they didn't know what they wanted to do they never got in-depth on anything. They never really put together the pieces that every single topic in electrical engineering is inexorably linked to every other. Analog circuits mean differential equations, considerations of bandwidth, frequency response, frequency content of waveforms, Fourier Series, linear algebra, matrix equations and any other number of fields of study. The coursework isn't a checklist, it's a symphony of learning. But if you don't have passion it is just a checklist. Mom and dad want you to be an engineer. You're smart, so you do well in classes and you graduate with a high GPA. You go for a job at a big corporation and they grind you into whatever kind of employee/engineer they want. Yay for you - you're average.

But the ones who have passion and drive and love what they've done - they stand out immediately as well. They had a definite plan when they went to college. They'll tell you how they took apart TVs when they were a kid (good Lord it's dangerous - let your kids do it but make sure those capacitors are discharged) or how they wrote dumb little computer games in Visual Basic to entertain themselves. They just won't shut up about their senior project or whatever personal projects they have if they haven't started their senior project. Their eyes light up when you suggest ways they could improve their project ('Ooooh my God... I wish we could go back and work on that more - I'd make it so much better!') or they kept working on it themselves after they graduated. They've learned weird programming languages for fun. In essence, they love what they do and they just won't stop doing it. They don't ask you for direction, they tell you - I'm this kind of engineer and I love doing it, do you need me? And the answer is usually yes, we need you.

So essentially the choice is yours: You can be average or excellent. There is certainly a long road between the two, but you have the choice to take the journey and practice practice practice. And if you find out that whatever you had your eye on doesn't really interest you then fine - move your target, pick something else. Excel at something. Hell, maybe you'll have 10,000 hours of random junk you've practiced. That's okay - it makes you an excellent generalist. Don't just sit around and play video games - ply your craft. I guarantee there's a payoff even though it's a long way down the road. Yes, a very long way. But it's worth it.