The cost of procrastination.
If you’re a regular reader of this blog, one thing you know is that I’ve been having chronic problems with the Start button on my 1995 Williams’ pinball machine, Johnny Mnemonic. If you’re new to the blog you may be shocked to know that (a) I own a pinball machine and (b) that anyone bothered to make a pinball machine about that terrible movie. Yes, it was a terrible movie, but the pinball rendition of it happens to be fantastic, and since I’m the biggest William Gibson nerd that ever lived, the purchase of it was pretty much a no-brainer for me. I’ve owned this monstrosity for several years now, and if you’re considering buying one of these things, I have two pieces of advice for you. First, they are a lot bigger than you think they are. Most of us have only ever seen pinball machines in commercial spaces: bars, arcades, movie theaters, etc. Those are large spaces and large machinery does not look large in them. Large machinery does look very large in your home. In fact, you should check your interior ceiling height to make sure it will even fit. In some lower-ceilinged rooms, it may not. The second piece of advice is that you need to understand that you are not buying a toy, you are buying a hobby. You are buying a perpetual project. Unless you buy a brand new $9000 machine from Stern or the new boutique makers, you’re buying a 30+ year-old electro-mechanical monster that was only meant to last a few years. That means there is constantly something needing fixing on them. They are very much like owning a classic car. You have to enjoy working on them, or you will not enjoy owning them. All that said, the great thing about pinball machines is that they are commercial coin-op machinery. These are not consumer-grade appliances. They are built like gorram tanks, and they are built to be maintained. We’re so accustomed to disposable things these days that it can be easy to forget what something built for maintenance looks like. I talk about this every time I write about Johnny, because it is still one my favorite things about these machines. Every single transistor is meticulously documented, and you can pretty much rebuild the entire machine with a screwdriver and a couple of wrenches. They were built to be maintained by cranky old operators driving around in their vans emptying quarters into a pail and cursing out the teenagers that kicked the coin door again. They are also built assuming teenagers are going to kick the coin doors. They weigh many hundreds of pounds (close to 1000 in some cases) and can take a remarkable amount of abuse. They are also built to fail gracefully. If a particular mechanism ceases to function, the software can route around it and keep the game operational (and earning quarters) until the operator could get to it for repairs. Almost any part on this machine can fail, but the game remains playable. Almost any part. You know what one part is really important? The Start button. Guess which part my game has chronic problems with. Well, you don’t have to guess, because I told you in the first sentence. Literally the first sentence. Keep up Quinn, geez.
If you look back on my Johnny series, you’ll see multiple repairs of the start button circuit. You’ll also notice a theme- I’m constantly doing the minimum that I feel like I can get away with to get the game operational again. That’s not generally my style, but for some reason I keep doing it in this case. I’m about to (probably) do that again, but we’ll get there.
The good news is that I didn’t even have to diagnose anything. I’m normally strongly opposed to starting a repair before doing diagnosis to confirm the problem empirically, and most Johnny posts start with a long section about that. In this case, I have fixed this damn button so many times that I knew exactly what was wrong. To be more precise, I knew exactly what was wrong because deep down I have always known and I have been procrastinating on the correct fix. As you may have read in previous posts in this series, I have traced all the wiring, checked matrix diodes, and all other sources of malfunction between the start button and the CPU board (where the start signal ultimate leads). All are fine. The problem is, and always was, at the connector. Let’s look at why.
The next step is to call in the supervisor and get her opinion.
My finger is on the problematic connector. Right off the bat, you can see that it’s a newer style than the others. This is a Molex crimped-pin connector, and the others are all IDCs (insulation displacement connectors). That’s because I replaced this connector previously. The reason this connector is so problematic is also visible in the above photo. See all that nasty staining on the metal bracket in the lower left? That is battery corrosion. In an amazingly poor moment of design, the backup battery for the game is factory mounted double-decker on the CPU board, using plastic standoffs. You can see two them, left of my finger there. That places the batteries directly above connector J212. Guess what batteries do sometimes? They leak. At some point in this machine’s history, batteries have leaked alkaline goo on to that connector, causing corrosion and the subsequent unreliability that I have been living with ever since. When I got the machine, everything looked okay and things were working. The batteries in it at the time were in good shape. Nevertheless, the first thing I did was relocate the battery pack to the side, as you can see in the above photo. That way, if they leak, they can’t hurt anything. I also replace them annually, but in case I ever forget, I’m protected.
It’s maddening that the batteries were not only factory-mounted in a place where they can cause damage, but they are literally mounted over the one connector that paralyzes the game if it becomes unreliable. Amazing. In any case, a couple of years ago I had replaced that connector, because the corrosion was apparent on it. That made the start button reliable again for a couple of years. Recently, it has been acting up again, but I have been able to keep it working by periodically cleaning the pins on that connector. The thing about battery corrosion though, is that it’s like circuit board cancer. It gets into solder joints and on traces, and it will slowly spread. It can be virtually impossible to get rid of once it gets into certain places. In some extreme cases, I have seen retro computing enthusiasts cut away sections of circuitboard and stitch-in new pieces to get rid of corroded areas. In a multi-layer board, if the corrosion gets into the inner traces, this may be the only option. If you have any old electronics you care about, do me a favor and remove the dang batteries. More than a few beautiful old computers, arcade games, and pinball machines have been ruined by this.
In my case, since the connector was new and the problem was worsening again, I knew it was time to look at the male side of the connector, on the CPU board itself. It’s a bit of a job, which is why I have been procrastinating on digging into it. I could procrastinate no longer, because my old tricks of cleaning pins and reseating that connector were no longer working. Johnny was quite dead this time.
The first job is to get that board out of there. There are a lot of connections on it that need to be carefully removed. Most of them have probably not moved since they were installed 25 years ago, so they may be quite stuck. The easiest way to remove them is to grab the wiring and yank really hard. Kidding! Kidding! Don’t do that, of course. Some care with thin bladed screwdrivers, picks, and gripping the connector’s body with small pliers is sufficient to ease them all off. You have to be mindful not to bend pins, so you have to work them side to side carefully such that they come straight off. If you bend pins on the connectors, you can straighten them again…. usually. If the metal is old and brittle, or has been flexed before, perhaps this is the time it snaps. You pays your money and you takes your chances.
There was a day when I would have meticulously labeled all those harnesses before removing anything. Experience has taught me that, when removing a single board like this, that really isn’t necessary. Those harnesses have been sitting in those positions for so long that they hardly even move when you disconnect them. It’s really easy to see where they all go back on. Even orientation is obvious because of the “set” that the wiring takes, but if a connector isn’t keyed, then it doesn’t hurt to mark one end of it. Most of them are keyed, though. Again, the machine is designed for maintenance.
Speaking of designing for maintenance, note that the PCB is held in place by screws, but they are key-holed, not just simple through-holes. This means you can loosen the screws and simply slide the board up to remove it. Why is that a big deal? They are mounted vertically in the back box of the machine, and below them is a gaping hole into the huge underbelly of the machine (itself filled with high voltage stuff). Dropping a screw in this situation would be supremely annoying at best, and fatal to the machine at worst. The simple design choice of key-holing the mounting spots removes all those issues in one swoop. It makes removing and reinstalling the boards quick, easy, and low risk. See? Design for maintenance. It’s the little things.
Okay, let’s get this board over to the electronics bench and take a look.
The top is about what I expected. Some corrosion on those pins, but debatably not enough to interfere with function. Let’s take a look underneath.
The underside of J212 actually looks fine. The joints all look okay and no corrosion is visible. There’s another area below it (in blue) that is showing some pretty lousy solder joints though. These look factory, and frankly there’s a whole bunch of these connector pins where the solder is a little light. As you probably know, a pin soldered to a pad should have a nice fillet of solder around it, like a little volcano. A whole bunch of these are flat- there’s barely solder bridging the pin to the pad. I don’t know if these were wave-soldered or done by hand. Knowing a bit about pinball factories, especially from back then, “by hand” is a likely answer. They built everything on these machines by hand, and largely still do. Check out online factory tour videos of Stern some time- it’s very much still a by-hand business. But then, look at most Chinese factories- a lot more of our products are still “hand-made” than we think. We don’t think of them as such if they are made on an assembly line by people in other countries, but an iPhone or your microwave or whatever are all still mostly made by hand. Even the circuit boards themselves. Sure, pick-and-place machines and ovens do most of the SMD work, but a lot of it is still drag-soldered by hand, as are all the through-hole components (unless the entire board is through-hole and can be wave-soldered).
Okay, I got side-tracked there. The solder joints look okay, but I think it’s well worth pulling J212 anyway. Time for some desoldering of very old components, which takes a bit of finesse. The first step, ironically, is to add solder.
Heat management is particularly important when working on vintage boards, because FR4 circuit board material is a lamination of fiberglass, copper, and epoxy. Epoxy is destroyed by heat, and the older it is, the more sensitive it is. If you overheat an area of the board, the epoxy in the substrate lets go and you’ll lift pads off the board or delaminate the area. New boards can take a lot of heat, but on a vintage one you need to get in, do your business, and get out.
Old solder doesn’t liquify or flow well, so adding new fresh solder smooths the rest of the process. We’re about to get out the big gun (sorta literally), and it doesn’t work well on old, brittle solder.
The Hakko is without a shadow of a doubt the most effective and most pleasant way to get solder off a board. There is a bit of technique to it, though. There is a minimum amount of solder required to be there, so it helps to blob on a bunch. It’ll suck up a huge blob with zero effort, but it won’t touch a thin film. You also have to learn to be patient with the vacuum. Give the heat a moment to do its thing. The top rookie move with this tool is to get trigger happy with the vacuum. That makes the situation worse because it sucks up part of the solder, leaving- you guessed it- a thin film. Now you’re stuck. It’ll seem like the tool is ineffective because of this technical error. Put a big blob of solder on there, get the hot end of the gun in place, give it a second or two to really liquify the whole area, then the briefest touch on that trigger and magic happens.
Most of the time, after hitting all pins with the gun, the part will simply fall off the board. If it doesn’t, I’ll give each pin a little wiggle with a screwdriver or pick to make sure it can move. If it doesn’t, there’s a solder film down in the hole somewhere holding it. The urge will be strong to force it at this point, but don’t! You’re likely to tear a pad off the board, and then you’ve quadrupled the challenge of your repair. This is especially true of a 25-year-old board like this. As I said earlier, the epoxy in the FR4 is old and brittle, and the pads can fall off the board if you look at them funny. If you get a pin that won’t wiggle, add more solder back on to it and hit it with the gun again.
Alright then, with the connector loose, let’s get a look underneath!
I knew there would be corrosion in there, but I wasn’t expecting it to be quite that bad. Well, that’s good news in a way, because we know where to focus our repair efforts. The first step is to neutralize that corrosion. As I said earlier, battery corrosion is like cancer, and you have to stop the reaction or it will continue to eat away at the traces and inner layers of the board. A good tool for this is vinegar. A lot of people refer to this corrosion as “battery acid damage” and you even see people try to neutralize it with baking soda. I guess this is a failure of our education system, because the batteries say right on them: ALKALINE. They are basic. This is base corrosion, not acid. I suppose people think batteries are all acidic because car batteries are, and that’s a big part of peoples’ life experience with batteries? I don’t know how we, as a culture, got the wrong message on this. Anyway, you want an acid for this job, and if you’re not in a hurry, a mild one is nice. Vinegar works, albeit slowly, or something like CLR or Bar Keeper’s Friend work if you’re in a bigger hurry. The latter is oxalic acid and comes in powder form. Mix up a solution and you’re good to go. It’s amazing on calcium build-up on shower doors and such as well (hard water deposits are basic, but too tough for vinegar). I wouldn’t use anything stronger than home cleaning supplies on a circuit board that you care about. You could create all manner of new problems if you get in there with brazer’s pickling acid or sulfuric acid from pool chemicals, or even worse something like hydrochloric or nitric acid (all of which I have lying around because my shop is a wonderland of bad ideas).
By the by, a lot of people put circuit boards in the dishwasher to clean them. Yes, this really is a thing. Pinball and retrocomputing people do it a lot. I have never done it myself, but it appears to be fine. However, if you’re thinking that will clean this corrosion off the pads, you’d be wrong. Why? Because detergents are- you guessed it- basic. Most food is acidic, so if you want a soap to be good at cleaning it away, make it basic. This is also why toothpaste is basic. A little junior high school chemistry gets you a long way in this world. Probably the stuff after junior high too, but I dunno. I stopped listening shortly thereafter. I’m sure it was all great.
After soaking for a few hours, it was time to clean up the mess. The vinegar reacts with the corrosion and leaves a precipitate on the pads (not sure what it is, I ain’t no chemist) that we need to clean off so that solder can once again be used. Anything to be soldered must be clean!
To clean up the pads, I used a combination of scotch-brite, brass wool, and some careful scraping with an X-Acto knife. I do a final cleaning with alcohol and cotton swabs. Once this is done, you can re-tin the pads with fresh solder. I also use a generous amount of liquid flux while retinning the pads, because it has an excellent cleaning action that will get any remaining fine residue off there.
Now I needed to deal with that connector. The pins had a fair amount of corrosion on them, and I was concerned that if I reinstall that connector, the corrosion will simply spread again. I needed to replace it. By a stroke of luck, I had that exact style of connector in my junk pile, except it was two pins too short (of course). However, I had two of them! I hacked one up and made the connector I needed from the pieces.
If you’re wondering why there is a pin missing in the original connector, that’s a key. I clipped the same pin on my replacement connector. Now the best part- reassembly!
Looking good so far. Time to put it back in Johnny!
Okay, moment of truth time, did I fix it? Time to power up.
You know, I really should have realized what was going to happen when I disconnected the batteries and pulled the CPU board. Of course that would mean I’d lose all the settings. More importantly though, I lost all my high scores. Had I thought about this, I would have left the battery pack connected to the board throughout the repair and it would have likely retained everything. I’ll be honest, I was genuinely a bit sad about this. You see, I had one very special score in there. Modern-era pinball machines like this have something called a Wizard Mode. It is so named because the The Who pinball machine was the first to do it, and it was named for Tommy The Pinball Wizard (of course). A Wizard Mode is like a final “mission” in a video game. After you complete all the tasks on the playfield, you get access to this special mode. It’s how you “solve” the game, basically. Johnny Mnemonic’s Wizard Mode is particularly epic, and is one of the main reasons I love the game. It’s called Power Down. When you start this mode, the game actually physically starts powering itself down in sections. For real. You have to hit certain shots repeatedly to keep the “battery” charged and keep playfield sections from shutting off. If you lose an area, then the pop bumpers go dead, it gets dark so you can’t see what you’re doing in that area, gates that you need get stuck closed, etc. It’s an incredible fourth-wall-busting experience the likes of which is rare in pinball. Why am I telling you all this? Because I have reached and completed Power Down exactly once. It is difficult and I’m frankly not that good at pinball. I was so proud of that score that it was my avatar photo on Facebook (back before it was evil and I was still on it). You can see that score in the first photo in this article. Why? Because I waited for the attract mode to show the score before taking the photo so you would see it. That score was the greatest moment of my pinball playing life. And it’s gone.
It’s okay- I did it once, so I can do it again. It’s a bummer, though.
On a lighter note, the game wouldn’t start yet, because it demanded that I set the clock. Why is that funny? Because I have owned this machine for almost 10 years and I literally had no idea it had a clock in it. I’ve never pulled the batteries before, and it doesn’t show the time in the game anywhere. Why does it have a clock? I suppose because the game has internal financial bookkeeping intended to be used by operators. I suppose it date-stamps that data, I’m not sure. It’s a whole region of the menus I’ve never really ventured into because it’s really dry and when you’re doing that you are not playing pinball. Digging through financial menus on a 4-line screen is a lot less fun than playing pinball, I can tell you. In any case, it won’t boot until you set the clock.
I had no idea how to even set the clock. It was buried waaaay deep in menus that I didn’t know were there.
Now we get to the real moment of truth! I set it to free play (it defaults back to quarter play and I don’t even have coin mechanisms in this machine) and pushed the start button.
Well. That’s disappointing. However, I was still extremely certain that the corrosion on that connector was the problem and I was very confident in my repair. So, I wiggled the connector a bit.
To quote the Buffy The Vampire Slayer musical, “The battle’s done, and we… kind of won?”. The start button now works, and appears to work reliably. However, it didn’t until I jiggled the connector a bit. That’s a trifle worrisome. It may be that there is corrosion inside that connector, picked up from the pins. Ideally, I should repin that connector to be sure. However, remember how I said that the harnesses in these machines are all exactly the right length? That’s no joke. I re-pinned that connector once before, and you lose some wire length every time you do that. There was barely enough wire to do it once. To do it again, I would have to splice all those wires and make a big mess of the harness. Is this more procrastination on my part? Perhaps. Or perhaps you have to draw the line somewhere. I’d already spent the better part of a day on this repair, and maybe the connector is fine. Maybe there was grit or some leftover flux in there. I’m going to leave it be, and if it gives me trouble again, I’ll have plenty of chance to catch it before the corrosion spreads back down into that connector. At least… I hope so.
Dun Dun DUNNNN
Anyways, that replacement Power Down score ain’t gonna earn itself. Gotta go play.
