Nerdy Texts of Analog and Embedded Systems Wizardry

So, I have a real thing for old Tek scopes.

This is due in no small part to the gospel preached by Jim Williams (you can get the cliff notes for said polemic here as well as a lot of other great scope-related stuff and some repair notes).

More personally, it’s also due to the fact that my first scope was a Tek — a 454 that I snagged from trash pile at my college job before I knew an opamp from an opcode. I learned how to use a scope by fiddling with that thing, and I broke it a lot being a careless kid (mostly in shipping). This meant I had to fix it a lot, too. My first job out of school was repairing stereo receivers and guitar amps, and fixing that 454 was both alien and awesome. Alien, because it was way more complicated than a Fender Twin, and awesome because the service manual was head and shoulders above any repair treatise I’d ever seen, whether it was for a piece of electronics, a car, or a piece of software. It really wanted you to understand the instrument. I loved that manual.

So later, once I started drinking the Williams kool-aid, it didn’t take a whole lot of convincing that the old Teks were “intellectual integrity” reified. I won’t bore you all with the details — Jim did it better anyway (as does Kent at his excellent site).

Anyhow, I bought an old Tek564b off Ebay a long time ago and had it sitting in the project pile. I got it because I was fascinated by the idea of the Analog Storage Oscilloscope. In a nutshell, an ASO allows you to save a trace on the screen of an oscilloscope by using some special phosphors in the CRT and a set of special electron guns. You can save any number of traces (two is easy, any more and it gets tricky) to compare or photograph. Since the stored output is not sampled, it essentially has infinite “bit-depth”, and it was built at a time when semiconductor memory was still on the drawing board.

This particular 564 and its plugins had a few problems, but one really stood out as a great example of the interrelated-ness of systems inside electronic equipment. I made the following video detailing this problem before I fixed the scope. At the time, I didn’t know what was wrong with the scope but I had a pretty good idea. I’ll give you a hint — if you are thinking along the same lines as I was at the end of that video, you’re wrong :-)

In the coming weeks I’ll post the actual repair, but play along at home and see if you can figure it out.

The above is a set of scope traces from the three coils of a BLDC which is being commutated incorrectly. Read on.

The job that’s been paying my bills and keeping me away from artsy-fartsy circuits for the past six months involves making a set of these enormous robot doors for a Certain Very Fancy Person’s house. Each door is 13 feet tall, around 7 feet wide, and weighs 1500 pounds. There are 66 of them in said house, and more in the servant quarters(!?!). The circuits on board each door have to handle running an onboard air compressor (which regulates a pneumatic weatherseal) as well as keeping track of temperature to linearize the pressure sensors when the weather gets cold. They also have to charge and maintain sealed lead acid batteries. They have commutated power rails. They have to communicate over said power rails, and do so using an capacitively-coupled data slicer and a proprietary protocol I wrote. This protocol has to be robust enough to bootload the processor over. It’s a proper embedded systems job.

All this having been said, the great majority of the real estate on the PCB is dedicated to running the big BLDCs that make the doors move. This post is some observations particulars of said motors.

I’d run lots of standard DC motors in past, and built H-bridges galore, but this was my first encounter with BLDC driving. As such, I made some mistakes in the prototype. And I am going to lay those out for you, dear reader, such that you might learn from the fires of MY workbench.

The motors in question were Maxon EC-Max 30 series motors with a gearhead. They are small, powerful, and expensive. We decided to use BLDCs for this job primarily because of their size, but the fact that the reliability is generally greater (no brushes) than a standard brushed motor certainly was a bonus. The BLDC driver circuit needed to fit into the stile of this door, which was only 1.5″ wide or so. The motors could be expected to stall under certain conditions, so the motor drivers had to be more Conan than not.

Once that driver was designed and fabbed, the fun on the workbench began.

A quick BLDC review:
Brushless DC Motors are like normal DC motors — except the user has to handle commutating the motor on their own. This means, more or less, wrangling the electric fields in the motor at the right time such that the motor turns. A standard DC motor has a set of “brushes” (usually pieces of carbon) that squeeze against the moving part of the motor (the rotor). These are responsible for getting power from stationary part into the spinning part. As the rotor turns, the brush presses on the correct bit of the rotor to get the fields doing their thing. This is cool because it’s easy to drive and relatively simple mechanically. The BAD thing about it is that the graphite/carbon brush wears down over time, AND the brush-scrape nature of the contact makes a world of tiny electrical arcs inside the motor which look cool but kick off a ton of electrical noise (and audio noise, which in this case mattered).
In a BLDC, it’s YOUR job to energize a set of coils at the right time to keep the motor turning. There are three (usually) coils in the stationary part of a BLDC and a number of magnets in the rotor. The trick is to energize the right combination of coils such that you pull the right magnet to the right spot at the right time. If you do this, the motor turns smoothly. It’s a sort-of-not-really like timing a spark plug, if you’ve ever had to do that.
The trick is, you need to know WHERE the rotor is at any given time, such that you can know when to turn off the last set of coils and turn on the next set. There are lots of ways to do this, but a common one is with hall sensors which read the rotation of the motor shaft via some magnets on that shaft. These are usually conveniently built into the motor. In my case, they read with 60 degree resolution, which is plenty to keep the motor moving correctly. Often, either a piece of hardware or an ISR reads the hall sensors, and then energizes the correct coils. This whole process can be as simple as a look up table.

There are lots of documents on the internet about BLDCs, but there are relatively few about what happens with incorrect commutation. The interesting part is when you mess it up and try to figure out what went wrong.

First step, the datasheet: The Maxon datasheet more or less recommends that you buy the Maxon motor drivers and just be done with it. Our form factor, target budget, and my not-insignificant sense of pride ruled that out. It DOES have a commutation chart, but it’s pictorial and is not that clear about “active high / active low” and other minor details like that. So, I made some guesses and let her rip.

In the course of letting her rip, I saw all kinds of exciting (read: trainwreck) motor beahvior. I figured out how that Maxon ran, but I also made a handy set of rules of thumb about what BLDCs do when you drive them wrong. From this, I made some procedures with which I ought to be able to start out with any unknown three-phase BLDC with hall sensors and derive how to get the correct drive sequencing. It’s like this:

Observations:

1.) BLDCs _WILL_ turn with some combinations of incorrect drive. This can mislead the un-wary.

2.) BLDCs usually not turn if the commutation sequence is mixed up. I assumed this would always be the case. It isn’t.
When a BLDC has a commutation sequence which is out of order, usually you see it STOP at some position and lock up, drawing stall current. If you aren’t on a current limited supply, or your motor drivers are not like Conan, you may damage your drive section. This is because whatever coil you’ve energized is pulling the BLDC to the wrong place and then KEEPING it there, since another set of halls has not been triggered. This probably means you have your driving steps out of order (or are energizing the wrong combination of coils).

3.) The interesting part — the BLDC _WILL_ spin, maybe, if you have the commutation sequence right BUT offset from the hall sensors readings. Meaning, if you are always energizing your coils one step ahead of where they should be, the BLDC will pull the motor to the right spot, but it will draw a lot of power doing it AND you will not have very much torque. If this happens, you probably have your hall sequence right AND your commutation sequence right, BUT they are lined up incorrectly.

Corollary:

1.) If you aren’t sure what your hall sequence is, you can spin the motor with your fingers (or a wrench, or whatever) and read the correct order of hall transitions.
2.) If you aren’t sure what your commutation sequence is, you can try messing with the order of different valid coil combinations and running your motor OPEN LOOP. This means stepping through your coil combinations on a timer, energizing them, and seeing if the motor turns. It will turn with bad efficiency again, and probably draw a lot of current, but it will turn when you have the sequence right. Best to do this on a current limited supply.
3.) Once you have those two pieces of information, you can line them up and figure out the correct drive sequence. This sequence will be the one in which the motor spins with the greatest torque and the best efficiency in both directions (IE, it has to work when reversed, too).

An unexpected benefit of doing this is that you see all kinds of BLDC commutation problems. This is useful when you’re debugging a system later, and have to deal with broken wires or damaged drivers, and can guess symptoms much more easily.

Analog is sexy, we all agree, right? Embedded systems on the other hand, are full of lots of unglamorous problems. Filesystems, say. Inherently un-sexy.

But I think component variation is maybe the best, most un-sexy problem there ever was. The unsexy cherry on the diet sundae. Like, you HAVE to solve it if you are making lots of something or that thing as a population will suck, even though the one on your bench always ruled.

All the pots in WTPA2 are these custom Taiwan Alpha jobbies. There are two values, 10kA and 100kA. The VCO uses one of the 10kAs as a coarse control, and it sets the voltage into a current sink which in turn sets the frequency. I’d been messing with the op amps in this circuit to try and get some performance improvements and “all of a sudden” one of the DUTs didn’t work correctly. At first I figured it was the opamp change, but after a lot of measurement and desoldering and component testing, it turned out one of the 10k pots was really 11.4k. This was a greater than 10% variation!

I’d built a margin in for error, but this was above it, and the current sink was getting too high of a voltage. I tested a dozen pots or so from the bin, and all of them were much less off. Still, since one was off, probably another one could be as well. It could even have been a result of the soldering process. I actually bothered to do a DC simulation at this point (using qucs) and fiddled with the component values until they were all as off as I could imagine them possibly being, and then resized the scaling resistor that sets the upper range of the VCO. It was a really crappy annoying unsatisfying solution, because it means that MOST of the units will be operating at a slower maximum clock than they need to. But that one in twelve or one in 100 will work correctly. Serves me right for getting the cheap pots, but there you go. Margin. Component variation.

Least Sexy Problem Evar.

TB

So, after getting back to client work for a minute, I decided to try and nail the clock pulse shaping circuit problem with a more viable solution than throwing in an extra $5 op amp.
The problem with the original pulse shaper circuit was simply that it was designed with a function generator and not a 20 cent opamp in a RC oscillator. The idea was sound (I think) but the values were not.

The real problem is that the square-to-pulse converter has to shape two different clocks — it’s always driving the same IRQ pin, but it can be hooked up to WTPA2′s 4046 based VCO, or the LM358 based on a user switch. The 4046 is HC logic, and has really square edges. The LM358′s edges are not square, and their slew rate seems frequency dependent also. So, you could optimize components for one or the other, but not both. I did some bench tests to figure out what I needed to change to get this right.

Check it. Here’s the rising edge of the output from the VCO:

And the corresponding output from the pulse shaping network:

Since we aren’t changing the VCO, this is what we’re gonna call “normal”. The top trace shows a risetime of about 0.1uS (scope is 0.1uS/div, 2v/div) which is quite fast (50V/uS in opamp terms). The ringing here probably has to do with the long ground connection on my probe, and it doesn’t hurt anything except my pride. The bottom trace (the output from the pulse shaper) shows a clean low pulse which is about 6uS long total (2uS/div)
Now, here’s the LM358:

And the corresponding output from the pulse shaping network:

Waaay different! This is the LM358 at its best incidentally — tested at low oscillator frequencies. At higher clock frequencies it slews even more slowly.
The top trace is 10uS/div, and shows a rise time of about 25uS (it’s 60uS with the clock cranked up to 25kHz). Annoyingly, it has that characteristic LM358 style crossover mess. AND it only gets up to about 4v. The rise time is really what matters though, and it is orders of magnitude slower than the 74hc4046. The bottom trace shows the output of the pulse shaper, trying but not quite making it. That dip never makes zero volts and might last 0.25uS. This doesn’t consistently trigger our interrupt-on-change IRQ.

So, the question was what to do. I tested a TLV2462 opamp (my goto op amp for embedded stuff, made by TI, a tank) and it performed equivalently to the 4046, and the pulses worked great. It’s slew rate was rated at 1.6V/uS, which is about 5 times faster than the LM358′s 0.3 V/uS. So it was faster, but not by orders of magnitude. If I could find an opamp which cost about the same as the LM358 and had a better slew rate, that seemed appealing rather than trying to hack up a circuit on 300 already-fabbed boards. The question was how fast we needed to go.

I settled on three opamps for the test: The Microchip MCP6002 (0.6V/us), the Microchip MCP602 (2.3V/uS) and the Texas Instruments TLC272 (5.3V/uS). A few days later I had them all from Digikey. I tested the MCP6002 first, since it was the cheapest. (0.27 at quantity, as opposed to the LM358′s 0.20) Surprise surprise! It worked great on the first try.
Although I didn’t measure the rise time, it looked clean on a scope. The ouptut from the pulse shaper was 6-7uS which is as good as (and more importantly in line with) the logic chip in the VCO. This was also consistent with the TLV2462.

In conclusion, the cheapest and easiest way to solve this problem is (I think) to eat 0.27 per kit and throw in another opamp. Further, the results are interesting because they show that above a certain rise time, performance remains the same. My guess is that there’s a knee point in that filter, and as long as the dominant frequency of the edge is above it, we’re good to go. In this case, a clean 0.6V/uS output was enough to trigger the shaper reliably.

Now that the results are consistent and I’m in tweak mode anyway, I’ll probably try and get those pulse times down by half or so, just in case the ISR gets faster.

Analog is fun, yo.
xoxox
TB

Last week was hellish.
Srrsly, yo. I forgot how much work this is. I flew my buddy Nick out from Chicago to be in charge of kitting and assembly, and my job was to get the firmware rocking. We had from June 20 to June 24 to stuff 300 kits, 100 jack boards, 100 drilled and tapped enclosure kits, build and test 100 microSD daughterboards, build a dozen assembled units, and get ready for Bent and our Solid Sound panel talk with Moog!

Woes, take 1:
China called and were like, yo man, your main boards are gonna be late. I blame myself for letting it get so close to the wire, and to be fair they were totally sports about shipping the paste stencils and small boards early. Still, with no main boards, I would have nothing to show at the festivals. Eff that. So I called up Advanced Circuits and was like, hook a brother up in the meantime, and they were like BLING BLING. So, I got 27 “Limited Edition” green pcbs, and made some acrylic enclosures to match. Financially, it was retarded. But I have my pride.

Woes 2:
Joe at Prototope really nailed it cutting a ton of enclosures. T&T PlasticLand over by Canal also came through in the clutch with like 100 pounds of fluorescent acrylic with prices that McMaster can’t hang with. However, some dumbass specified that all these enclosures should be drilled and tapped, and those operations alone took DAYS, even with my fancy drill jig:

Here’s Nick hating life:

Woes 3:
That effing pulse shaper circuit (see the last couple posts) was wrong. Of course we didn’t figure this out until an hour before Bent. It was borderline such that it worked _a little_ even though the circuit had not changed since the prototype. The routing and components (though not the component values) had changed, and that was enough. Basically the LM358 had shitty rise times into whatever load the circuit presented, and the effective edge frequency (what the pulse shaper really looks at) was too low to work. I threw a handful of expensive TI opamps into some kits and dragged them out anyway, determined to have something to sell, but I only thought of this after Bent (but before we drove to North Adams for the festival). The new opamps slewed a lot faster and were an effective (if again, expensive, bandaid).

Woes 4:
The microSD card. I came up with new swears for these things:

Originally for this project I bought a crappy Kingston 2GB uSD card for testing from a pre-paid cellphone store near my house. FOR WHATEVER REASON, it turned out to be the fastest, most forgiving device ever. This week, on a whim, I ordered every crappy uSD card between 512MB and 2GB that I could find on Ebay. They all behaved differently. It took days to test my drivers to make sure that all the cards behaved correctly, and there are definitely exchanges in there that you have to do which have pretty much zero to do with the SD spec (or at least the free one). This sucked, to say nothing of then trying to make a filesystem and buffers to read audio in realtime. While card access was rock solid for all tested cards by Bent, I kinda though my sample read-write routines sucked. In the end I threw them out. The devices at Bent could format an SD to the WTPA filesystem (which is NOT FAT16, but a more real-timey system that I think makes more sense) and that’s about it.

Woes 5:
Driving to North Adams after Bent with a trunk full of expensive, lovely, VERY PROTOTYPE-EY WTPA2s was the worst experience ever. I’d been up for about 72 hours on about 4 total hours of sleep (none the night before) and I seriously saw animals that do not exist in this world. Anybody who can’t afford bad acid should try writing device drivers for three days while inhaling plastic fumes and then driving through a woods full of deer at midnight.

But then we got there, pounded a bunch of beers with our nerd friends, got pocket protectors from eminent wizard Cyril Lance of Moog and generally had a great time.

And, oh yeah, in the process we made THIS:

Bent looked like this:

Shop aftermathz:

WTPA2 is not ready to sell, but I have 300 of them and they’re pretty f’ing close. Expect to see the sales link by the end of July.
TB