Tuesday, February 27, 2007

What was that?

When I was a young lad I had, shall we say, a more "carefree" attitude towards software development. This is the story of how I learned to be a bit more methodical in my approach to certain types of issues.

The project I was working on was the motion control system for a robot. Now, I should probably clarify that, so you don't get the wrong idea - we're not talking about C3PO or R2D2, here. This was an industrial robot, used for 3-d imaging. The robot itself was made out of slabs of cast iron bolted together, was probably 8 feet tall, 4 feet wide, and ten feet long, and weighed in the neighborhood of 4 tons.

The problem I was having was that the motion control was somewhat unresponsive - you'd move the joystick, and the translation table or the optical head would slowly start to move, and when you got to where you wanted to go, it'd keep on moving for a little while before coming to a stop.

As I was looking at the code, I found what I thought was the problem - I had simply put the wrong coefficient in for one of the control equations, so we weren't getting the proper exponential factor applied to the requested motion. A quick edit and recompile, and I was ready to test the new code.

I started the system up, and very slightly moved the joystick. The translation table started to creep forward. I then pushed the stick over a little farther, and the table accellerated. So far so good.

And then, something very bad happened. When I put the stick back to the rest position, the table didn't slow down. In fact, it kept speeding up. I tried pulling back on the stick, but that didn't seem to have any effect. I managed to turn off the power to the motors just before the table hit the hard stops at the end of its travel.

So this 600 pound cast-iron table slams into the rubber bumpers at the back of the machine, going something like 30 feet per second. The whole machine rings like a gong, and all work in the entire shop grinds to a halt as everybody looks over to see this multi-ton machine gently rocking back and forth. I was really worried that I'd wrecked at least part of a very expensive machine, but a later calibration run showed that the mechanical parts of the robot were just fine.

It turns out that there were two problems in the system - the incorrect exponential on the input side that I'd corrected, and an additional incorrect damping factor on the output side. The upshot of all this is that once the system was up to speed, it took a very long time to slow down, but the bug on the input side ensured that it never got up to more than a tiny fraction of the maximum speed.

The next time I needed to make a change in those calculations, I did a "dry run" with the motors disconnected first...

6 comments:

Anonymous said...

... I looked at my kingdom, I was finally there, to sit on my throne as the Prince of Bel Air!

Mark Bessey said...

You know,I was just going to delete this as a spam comment, but now I'm intrigued. Who would post a "Fresh Prince of Bel-Air" quote into a random blog?

It's just too weird to be a drive-by comment. Someone must be trying to tell me something...

Harlan Iverson said...

Random comment = the story is on Reddit: "This is the story of how I learned to be a bit more methodical in my approach towards software development."

http://programming.reddit.com/info/1ap93/comments

(if you hadn't figured it out yet)

Anonymous said...

With something that massive, shouldn't there be a code object that manages safety and resides prior to the direct motor manipulation???

Mark Bessey said...

Yes, there are a lot of things I would do different if someone gave me this assignment now.

Amongst other things, having a way to test everything *except* the motors would have been a good idea.

When we delivered the system, there was in fact a simple piece of code that enforced absolute maximum speed limits on all the axes. It also automatically shut the system down with an appropriate error message when commanded to do something really stupid, like push the table through the stops or smash the optics into the table...

Unknown said...

Probably a reference to the theme song from Fresh Prince, which I memorized in the heady days of my youth. It starts:

This is a story all about how my life got flipped, turned upside down and I'd like to take a minute just sit right there and tell you how I became the prince of a town called bel aire.

Bears a passing resemblance to your opening line...